Slot Memory [TM]

rtfmoz · Post by **rtfmoz** » 2003-03-11 00:25

Well not trademark really but (this has probably been presented a thousand times before) here goes...

A real problem with DC++ is it does not remember jack shit. I propose the following modifications and I will write the patch myself. I just want to get some feedback from the developers on the idea.

A client wants to get a file. It generates a random large number as a key for each file and saves this to disk as part of the request queue. The client then requests the file with its key. The server saves the client request with the key and if a slot is free starts the upload to the client. If a slot is not free the server queues the file in its upload queue and saves this to disk.

If an upload ends prematurely, the server waits a few seconds and then starts the next file in the queue. If previous queued client returns to the server and presents the correct key the server will close most recently started download and resume the previous one. If a clients connection is closed during a download it should retry at normal timer interval. Inactive queued downloads expire in seven days
.
rtfmoz

sarf · Post by **sarf** » 2003-03-11 10:35

Ehm... what is the problem that this solution solves?

If it is an upload queue you are looking for then the client might remember the IP and nick of the user instead.

Sarf
---
Everything you know is wrong!

rtfmoz · Post by **rtfmoz** » 2003-03-11 19:39

Well the idea is that it remembers the people who have queued a download. DC++ does not do this presently. It can tell you who is waiting but not in any specific order. This will queue them in order and if they go away and come back it will continue their download if the file was incomplete. It also prevents people hijacking queue positions by using a keyed downloads known only by original the server and client. IPs change and so do peoples nicks. This way they dont lose their queued downloads.

rtfmoz

rtfmoz · Post by **rtfmoz** » 2003-03-11 19:51

Let me add some more on this, when they initially request the file DC++ should remember the username, filename, and key and their order in the queue if no slots are free. If the user is not online when a slot is free it skips them but keeps their place in the queue. If they try to connect again with the right key and filename and the are in front fo any exisitng downloads it will close the most recently started download and resume the one with the user.

sarf · Post by **sarf** » 2003-03-12 12:20

The whole upload queue thingy can be solved by "reverse connecting" (an idea that someone on the boards submitted) to the client who is supposed to get the slot.

More information about this can be found by searching the boards. It's a very neat way to solve the problem with backwards compatibility, since only the client which does the uploading needs to use it.

Sarf
---
Time's fun when you're having flies

rtfmoz · Post by **rtfmoz** » 2003-03-12 15:36

It doesnt really,

Only the downloading client knows when its ready to connect and not the uploading one. So if I go offline and my slot comes up then what happens? I lose the slot or does the uploading client keep polling for me? That does not sound very practical.

My way the client can re-identify itself as the owner of a queue position if the IP and/or Nick has changed because it will have the original key and full path for that download. The idea is if they reconnect and request the file using the same key it will match them up with their original queue position if it still exists. If they are in front of any of the present downloaders then it will start their download immediately and close the mos recent download.

The idea is to preserve the first in first served order which IMHO the only way that is fair.

rtfmoz

Post by **GargoyleMT** » 2003-03-12 22:43

Ummm, sarf is right.

And what you're describing seems to be an upload queue, except perhaps persistent across runs of DC++. I personally think that if a client wants to download from me, and they go offline, then they forfeit their spot in line.

BCDC++ already has this, though I haven't had the rationale behind the half-connect explained to me, since it seems that a normal $ConnectToMe would work just as well.

rtfmoz · Post by **rtfmoz** » 2003-03-12 23:54

Well thats is total crap.

This *is* an upload queue proposal. But when I want to use my connection I turn off DC++ because it chews too much bandwidth. Why should I lose my queue slot? I was their first mate and frankly, if I come back, I would like the slot back. Some of us dont live online 24x7. Its only fair.

rtfmoz

rtfmoz · Post by **rtfmoz** » 2003-03-12 23:55

And bcdc++ queueing is not persistant hence my proposal. Sorry if I appear to be rude. Not intentional.

Post by **GargoyleMT** » 2003-03-13 07:50

rtfmoz wrote:But when I want to use my connection I turn off DC++ because it chews too much bandwidth. Why should I lose my queue slot? I was their first mate and frankly, if I come back, I would like the slot back. Some of us dont live online 24x7. Its only fair.

Well, it's the way things normally work when you stand in a line in real life. If you step out of line, you lose your spot. If you come back and try to get in the same spot, that won't be well received by people who were behind you.

If you don't agree, that's fine, but I think that's probably the opinion of many (right or wrong), and you need to account for that if you're going to implement this feature.

As an aside, if DC++ uses too much (upload) bandwidth, shame on you for not finding one of the technical solutions to the problem. My computer is on 24/7 (I assume you have broadband as well?), there's no excuse to turn it off because I like sharing files with people.

[A&AM]Fireball · Post by **[A&AM]Fireball** » 2003-03-13 08:05

1) if you want to save bandwith for something else it's your problem ...
2) users with lots of wanted files (>500 GB good share

will end up with a huge key file
3) someone with REALLY slow download speed (sometimes a connection is just plain slow, or some modem user) will block the slot virtually forever
4) ppl with a slot from someone with fast upload speed could (theoretically) give the key away bevore completing the file (getting the last few MB's elsewhere) and someone else could use the key to get a top position at that fast user's queue .. *urks*

rtfmoz · Post by **rtfmoz** » 2003-03-13 14:09

1) if you want to save bandwith for something else it's your problem ...

Crap. You live on a 512/128 connection and see if you still have that opinion.

2) users with lots of wanted files (>500 GB good share will end up with a huge key file

Yes but key files are not tha large in the first place and it expires over time.

3) someone with REALLY slow download speed (sometimes a connection is just plain slow, or some modem user) will block the slot virtually forever

Block the slot? There is no blocking, if they are not using it someone else takes it. If they are using it its theirs, no different to now.

4) ppl with a slot from someone with fast upload speed could (theoretically) give the key away bevore completing the file (getting the last few MB's elsewhere) and someone else could use the key to get a top position at that fast user's queue .. *urks*

Simple, dont tell em what the key is. Its under the hood so the user does not need to know. Dont say "they will modify it" cause they can modify it now.

rtfmoz

rtfmoz · Post by **rtfmoz** » 2003-03-13 14:23

Well, it's the way things normally work when you stand in a line in real life. If you step out of line, you lose your spot. If you come back and try to get in the same spot, that won't be well received by people who were behind you.

Thats a load of crap. If you go away to get something you leave a friend in the queue for you. If you have to do something you put a friend in the queue for you. Just because you are not always around it does not mean you should lose your posiiton. This is REAL life and time is limited. Using up all resources standing in a queue is a waste of time and money.

If you don't agree, that's fine, but I think that's probably the opinion of many (right or wrong), and you need to account for that if you're going to implement this feature.

I think a lot of people would welcome the benefits this feature would bring. If you connection dies for some reason, if you have to upload a file to someone and you need the bandwidth, if you cant leave your computer on 24x7, if you want to play an online game, you dont lose your position in the queue. There is NOTHING worse than downloading a 600MB file to find that your connection died while you were away and you have to wait another three days before a slot is free.

As an aside, if DC++ uses too much (upload) bandwidth, shame on you for not finding one of the technical solutions to the problem. My computer is on 24/7 (I assume you have broadband as well?), there's no excuse to turn it off because I like sharing files with people

Again thats complete crap. Shame on me? LOL I have tried all the technical solutions and they do not work well at all. My computer is on 24x7 but I cannot leave DC++ on all the time as it eats up the bandwidth of my connection. What about when I want to USE my connection? The world does not revolve about DC++! When I close it down I lose ALL my slots and position in any queues. Some people are charged by volume for their downloads so they cant afford to leave DC++ running all the time.

rtfmoz

sarf · Post by **sarf** » 2003-03-13 16:41

rtfmoz wrote:Thats a load of crap. If you go away to get something you leave a friend in the queue for you. If you have to do something you put a friend in the queue for you. Just because you are not always around it does not mean you should lose your posiiton. This is REAL life and time is limited. Using up all resources standing in a queue is a waste of time and money.

No... but then perhaps you should employ a "friend" application which does nothing but stand in the queues you are in. Preferably this should be done from another computer than the one you use so any reboots/et cetera would not affect the "friend" application.

Since the current slot gathering model of DC is accepted by all (if not liked) having an upload queue would by a tremendous step forward. I dislike the idea of a person going into a hub and requesting an upload queue key of every person on the hub, only to be able to download anything he/she might want later on. This is something I personally would recode the client to do, since it has a benefit for me.
Having a non-persistent upload queue is good enough, and ensures that people will not have hundreds of clients waiting for "their" slot.

rtfmoz wrote:I think a lot of people would welcome the benefits this feature would bring. If your connection dies for some reason, if you have to upload a file to someone and you need the bandwidth, if you cant leave your computer on 24x7, if you want to play an online game, you dont lose your position in the queue. There is NOTHING worse than downloading a 600MB file to find that your connection died while you were away and you have to wait another three days before a slot is free.

In the "three day wait"-case I'd recommend you to contact the person with the file you need and ask to be granted a slot. A non-persistent upload queue would ensure that, since it is non-persistent, there would be less people waiting for their slots ahead of you. If the file is that important to you, then I'd expect you to sacrifice something for it... such as time spent waiting to get a slot or spent trying to get a slot granted to you.

rtfmoz wrote:Again thats complete crap. Shame on me? LOL I have tried all the technical solutions and they do not work well at all. My computer is on 24x7 but I cannot leave DC++ on all the time as it eats up the bandwidth of my connection.

Ah... GargoyleMT meant, in translatation, to use an "upload limited" client or to make your own. You would thus be able to control your bandwidth.

rtfmoz wrote:What about when I want to USE my connection? The world does not revolve about DC++! When I close it down I lose ALL my slots and position in any queues.

Umm... there's an inherent contradiction here. If DC++ is not that important to you, then you should not care when you lose your slots/position in queues and so on... or is it merely that you wish to retain the good things without suffering for it?

rtfmoz wrote:Some people are charged by volume for their downloads so they cant afford to leave DC++ running all the time.

Again, an "upload limited" client could (and would) help them.

rtfmoz wrote:
[A&AM]Fireball wrote:2) users with lots of wanted files (>500 GB good share will end up with a huge key file
Yes but key files are not tha large in the first place and it expires over time.

Well, how much would the overhead for the "big sharers" be in CPU and I/O? If it is significant it should be known.
Let's say the key files are persistent for seven days. Let's say the big sharer is on three hubs, each with 500 users. Every day, the big sharer get 1000 requests. Of those, half is handled, the rest are queued (simplifying a bit here, of course people would get queued one day and handled the next, but anyhow). Over the course of seven days, the number of key files would grow to 3500, and be constantly at that size since new key files would be added as the old files are pruned.
Maintaining 3500 files, pruning them when appropriate and searching through them (not very hard, but...) would take some overhead... though mainly in the maintaining part.

Of course, this example was taken from thin air, but extreme cases should be considered since if they are not people would simply uncheck the "maintain upload queue" option (I hope you're not going to force people to use it). The other side would have to be considered too, since clients usually have a rather large number of other clients they may need to maintain a key file for.

rtfmoz wrote:
[A&AM]Fireball wrote:4) ppl with a slot from someone with fast upload speed could (theoretically) give the key away bevore completing the file (getting the last few MB's elsewhere) and someone else could use the key to get a top position at that fast user's queue .. *urks*
Simple, dont tell em what the key is. Its under the hood so the user does not need to know. Dont say "they will modify it" cause they can modify it now.

But this raises an issue... though not especially important - the key should be unique and perhaps only good for a number of files/amount of data/time/whatever? It'd be more of a ticket then, rather than just being a queue-number with bells on. Thus the uploading client could choose to use the rating (tm) of the user that wants data from it and handle the "ticket" accordingly, or it could use the connection type of the client, or the current phase of the moon or whatever. The point is, it could be used to minimize any abuse of the place it secures (the slot).

That's something that I could imagine being implemented... but there is still no real reason not to just implement this on the uploading client and work like a normal DC++ client for all other intents and purposes.

Sarf
---
Think "HONK" if you're a telepath!

Post by **GargoyleMT** » 2003-03-13 21:37

Sarf, as usual, you've done an excellent job of bringing up the issues that are involved in this line of reasoning. My only addition is below:

sarf wrote:
rtfmoz wrote:Again thats complete crap. Shame on me? LOL I have tried all the technical solutions and they do not work well at all.
Ah... GargoyleMT meant, in translatation, to use an "upload limited" client or to make your own. You would thus be able to control your bandwidth.

Yes, this is exactly what I mean. The simplest technical solution to me, in terms of work, is to use a upload limiting/throttling client.

rtfmoz · Post by **rtfmoz** » 2003-03-13 23:18

No... but then perhaps you should employ a "friend" application which does nothing but stand in the queues you are in. Preferably this should be done from another computer than the one you use so any reboots/et cetera would not affect the "friend" application.

Ok there appears to be some misunderstanding here. Probably because I am not very clear about it. The downloader only needs to present a unique'ish key with every file requested. Nothing else changes in regards to downloads. There is a different key for each file you want - a large random number will do.

The uploader takes that filespec and key as a unique pair that represents the download from that user. It stores it in a queue if their is no slots free. That queue entry will expire in seven days if no activitity has occured. By activity I mean that some data has been downloaded or it has been re-requested.

I dislike the idea of a person going into a hub and requesting an upload queue key of every person on the hub, only to be able to download anything he/she might want later on.

No-one can request a key. When a someone wants to download from you they present a key per file

ensures that people will not have hundreds of clients waiting for "their" slot.

This is how it is now! I am not changing anything I am just saying if someone is in the upload queue ahead of you on that specific user then they get preference!

In the "three day wait"-case I'd recommend you to contact the person with the file you need and ask to be granted a slot. A non-persistent upload queue would ensure that, since it is non-persistent, there would be less people waiting for their slots ahead of you.

No-one is left waiting. If the slot is not in use it gets used by the next person in the queue... and the next... and the next... until you return to continue the download.

Ah... GargoyleMT meant, in translatation, to use an "upload limited" client or to make your own. You would thus be able to control your bandwidth.

I know he did but so many hubs kick you when you upload limit so thats no solution.

If DC++ is not that important to you, then you should not care when you lose your slots/position in queues and so on... or is it merely that you wish to retain the good things without suffering for it?

Crap! When you see a problem you fix it. This is a problem and I am fixing it. What I like or dont like is irrelevant. The world does not revolve around the USA and many other countries cannot stay online indefinately. People with this issue are disadvantaged. It will not affect people who dont have this problem.

I mean come-on... You are telling me that uploading in DC++ is a completed feature? If you have a download queue then why haven't you implemented an upload queue... it appears that it has never been finished.

Again, an "upload limited" client could (and would) help them.

How does an upload limit help volume quota? It doesnt, the file is the same size. Turning off the client is the only solution that stops you using bandwidth.

Well, how much would the overhead for the "big sharers" be in CPU and I/O? If it is significant it should be known.

It should be one file. Probably a seperate queue.xml file with the upload queue information. Since when does a 1MB file cause any problems? As for the extreme case I see your point. Initially that may be so but things can be tuned as we progress.

Maintaining 3500 files, pruning them when appropriate and searching through them (not very hard, but...) would take some overhead... though mainly in the maintaining part.

Assuming it is one key file, efficient searching will make light work of this.

Of course, this example was taken from thin air, but extreme cases should be considered since if they are not people would simply uncheck the "maintain upload queue" option

I agree and I would never force people to use it.

ppl with a slot from someone with fast upload speed could (theoretically) give the key away bevore completing the file (getting the last few MB's elsewhere) and someone else could use the key to get a top position at that fast user's queue .. *urks*

I am sure we can some up with something...

But this raises an issue... though not especially important - the key should be unique and perhaps only good for a number of files/amount of data/time/whatever?

The idea is keys are unique per file requested.

That's something that I could imagine being implemented... but there is still no real reason not to just implement this on the uploading client and work like a normal DC++ client for all other intents and purposes.

I agree but since the upload and downloading client are the same thing then its a moot point.

Personal Opinion Below.

Christ!!! Do all you people have blinkers on or what? Are you people so stuck in a rut that you cannot see the obvious advantages. Please can you try and look at it from the advantages that it brings and discuss how it might work

rtfmoz

sarf · Post by **sarf** » 2003-03-14 07:04

rtfmoz wrote:[friend quote]
Ok there appears to be some misunderstanding here. Probably because I am not very clear about it. The downloader only needs to present a unique'ish key with every file requested. Nothing else changes in regards to downloads. There is a different key for each file you want - a large random number will do.

Ah. Now we're talking business here. FYI, what I meant was that comparing the queue system with the real world was doomed since we do not have a "friend" in some form, shape or color to use in DC++... though that cold be changed.

rtfmoz wrote:This is how it is now! I am not changing anything I am just saying if someone is in the upload queue ahead of you on that specific user then they get preference!

Yes. My point is simply that it is a) simpler to implement "uploading side" queue handling and b) ensures that it works for everyone - a system where both clients need to be of a specific version/type is less efficient (overall) than one that needs only to be supported on one end to work.

rtfmoz wrote:No-one is left waiting. If the slot is not in use it gets used by the next person in the queue... and the next... and the next... until you return to continue the download.

Yes, this was understood, but what was meant was the comment made about having to wait three days for a file if the other client used a non-persistent upload queue OR used the current "queue" system.

rtfmoz wrote:I know he did but so many hubs kick you when you upload limit so thats no solution.

Well, there are clients that do not report that you use an upload limiting client. For instance, you can modify the code yourself.

rtfmoz wrote:Crap! When you see a problem you fix it. This is a problem and I am fixing it. What I like or dont like is irrelevant.

Apperently it is not irrelevant to you, since it seemed to be something you felt.

rtfmoz wrote:The world does not revolve around the USA and many other countries cannot stay online indefinately. People with this issue are disadvantaged. It will not affect people who dont have this problem.

It will affect people without the problem because clients will be changing their behaviour - as a user of an unlimited 10-megabit line you can "afford" to hammer for a slot in the current system. I do not say that this is better in any way, just that it will affect others.

rtfmoz wrote:I mean come-on... You are telling me that uploading in DC++ is a completed feature?

I am not telling you that it is finished. I am telling you that, because of "political" reasons, arnetheduck will never include upload limiting in his version since it currently would mean that DC++ would be banned in a large percentage of hubs.

rtfmoz wrote:If you have a download queue then why haven't you implemented an upload queue... it appears that it has never been finished.

The reason? Because NM DC does not have one. The upload queue has never had been even started on in DC++.

rtfmoz wrote:How does an upload limit help volume quota? It doesnt, the file is the same size. Turning off the client is the only solution that stops you using bandwidth.

Ehm... with an upload limited client they could decide how much upload they wish to use, thus giving them more time online (since their volume would be used up at a rate they decide).
Besides, if the only thing that helps them is to turn off the client, then they will not be helped by anything you or I can do.

rtfmoz wrote:Assuming it is one key file, efficient searching will make light work of this.

Yes, thank you for clearing this issue. Since you were talking about key files I got confused.

rtfmoz wrote:The idea is keys are unique per file requested.

An important design decision is then how many files (and thus keys) each user is allowed to have at the same time on one particular client.

rtfmoz wrote:I agree but since the upload and downloading client are the same thing then its a moot point.

No, not when people are using NM DC / oDC / pDC / BCDC++ / blue / et cetera clients.

rtfmoz wrote:Christ!!! Do all you people have blinkers on or what? Are you people so stuck in a rut that you cannot see the obvious advantages. Please can you try and look at it from the advantages that it brings and discuss how it might work

I think it has advantages. I think it is more advantageous to do a pure "uploading side" client-fix since then it would work for all. I also think that having two ways of handling upload queues is a Bad Thing (tm).

Sarf
---
Sometimes a cigar is just a cigar.

rtfmoz · Post by **rtfmoz** » 2003-03-14 07:17

think it is more advantageous to do a pure "uploading side" client-fix since then it would work for all. I also think that having two ways of handling upload queues is a Bad Thing (tm).

OK you lost me here. When you say a pure "Uploading side" I am thinking a modification of DC++ that adds the upload queueing functonality. Now as for the two ways of handling upload queueing, I am not suggesting their will be two ways.

If you have a upload queueing turned on then DC++ can fudge a key if the downloading client does not present one. This means upload queueing still works regardless if the other end presents a key or not. They dont however get the advantages of having presented their own key.

rtfmoz

rtfmoz · Post by **rtfmoz** » 2003-03-14 07:21

Yes. My point is simply that it is a) simpler to implement "uploading side" queue handling and b) ensures that it works for everyone - a system where both clients need to be of a specific version/type is less efficient (overall) than one that needs only to be supported on one end to work.

I agree totally! The only problem I see is highjacking of queue slots hence the idea of the key system. If it is simpler to just used nick then that can be the basis for the queueing but it is open to abuse. I was suggesting a *key* would prevent this type of abuse.

rtfmoz

[A&AM]Fireball · Post by **[A&AM]Fireball** » 2003-03-14 07:26

i'm on 768/128 ... and i still have the opinion

Post by **GargoyleMT** » 2003-03-14 07:43

rtfmoz wrote:I agree totally! The only problem I see is highjacking of queue slots hence the idea of the key system. If it is simpler to just used nick then that can be the basis for the queueing but it is open to abuse. I was suggesting a *key* would prevent this type of abuse.

If this is the problem that the key solution is intended to fix, there are other ways of approaching it. For instance, clients check in every so often (50/60 seconds) to see if there is a new slot. If someone has gotten offline, or hasn't checked in in a specific interval, you could see if someone's "entry" time into the queue matches up, and if they have the same IP or are in the same IP block.

I don't think that nick changing would be worth, if it was my time doing the programming, implementing the whole key solution. Instead, the fix that I just mentioned would handle most of those cases, with only a little complexity on the uploader's system.

The issue of how many keys a client can have in an uploader's queue is a scenario not likely to be encoutered much in the real world. DC++ is deterministic in how it picks which file to transfer, unless you're constantly changing priorities of the files in the queue (or adding new ones with highest priority), it should present the same file all the time. Right...? (Check the code.)

And thank you for making clear you were talking about per-file unique IDs. It sounded like you were just giving out IDs for a slot. Not doing so makes your system much less abusable.

HaArD · Post by **HaArD** » 2003-03-14 08:49

I've read through this because some sort of "Who's trying to download from me" queue inside the client would be great. but, as has been stated it must be compatible with all clients out there. This necessitates it being a Uploading Client side solution.

I grabbed the latest BCDC and it does show who's waiting but I can't find any more information about how it works/what it does. Can someone explain that or point me somewhere?

I've done some testing within my hub, using a script and I think I know how it works... it listens to $ConnectToMe and $RevConnectToMe and if they continue to come (every 1 or 3 minutes from my testing) then it adds that user to the Q.

Is priority given to the first person in that list when a slot becomes available? The way DC++ works now it's the first person who sends a connection request, which means that $ConnecToMe/$RevConnecToMe flooding IS an effective way to increase your chances of getting a slot, which is not good.

I like the idea of having a queue, so I can see and manage who's in line, I'd like to be able to"

Re-order the Q
Bump someone to the top of the list
Grant a slot but only for 'n' specific files (ever Get a PM "Can you grant me a slot so I can get the last 3 files of..."?)

I totally disagree with the idea that the place in line should be held for the downloader if they decide to go away for the weekend and shutdown DC. Your place in line can be held, while you are in line, you leave = you lose.

Having people stay online in the hubs is a good thing for DC, creating a situation where disconnecting has no 'cost' goes against that.

I understand the u/l limits of users and the bandwidth cap's and the asynchronous connection issue and therefore I allow bandwidth throttling in my hub and have supported it being included in the client.

Unfortunately the politics of this issue are well known, it has been debated many, many, many time... the Swede's with their subsidized 10mBit connections couldn't care less about the rest of us and don't want b/w throttling, so it'll never happen in DC++. C'est la vie. Find more reasonable hub-owners that will allow other clients.

Tolerance should be built in of say 10-15 minutes to account for power failures/system crashes/hub disconnects etc. This should be "persistent" ie written to disk, because it might be my machine that crashes and I want to give the slots back to the people who had them before once I am up and running again (It's only fair...)

I would not want some huge file of every leecher who Q'd half my share on every hub I every popped into for the past 7 days sitting on my machine getting trimmed one file at a time...hugely inefficient.

But, that's just my opinion...

HaArD

Slot Memory [TM]

Slot Memory [TM]

Re: Slot Memory [TM]