Why segmented downloads are a horrible idea

Use this forum to flesh out your feature request before you enter it in <a href="http://dcpp.net/bugzilla/">Bugzilla</a>.

Moderator: Moderators

Locked
lordadmira
Posts: 52
Joined: 2003-03-12 00:06
Location: Zinzinnati

Why segmented downloads are a horrible idea

Post by lordadmira » 2006-01-28 19:11

Below is a chat log in which I explain why segmented downloads are a horrible idea. (I am bob)


[02:25] <[VIP]Zaszâ„¢> segmented downloads
[02:25] <[VIP]Jamgoggâ„¢> so you can download what other are downloading even if their download hasn't finished
[02:26] <[VIP]Zaszâ„¢> basically means, dl a file from multiple users at once and that as well, i think
[02:26] <[VIP]Jamgoggâ„¢> oh, so kinda like a torrent
[02:27] <bob> segmented downloads are annoying
[02:27] <[VIP]Jamgoggâ„¢> why is that
[02:27] <[VIP]Jamgoggâ„¢> bob
[02:27] <[VIP]Zaszâ„¢> i like that
[02:27] <bob> each chunk makes a log entry which fills up ur hard drive
[02:27] <[VIP]Zaszâ„¢> not mine
[02:27] <[VIP]Jamgoggâ„¢> so turn of logging
[02:27] <[VIP]Jamgoggâ„¢> off
[02:28] <bob> and using segments doesn't improve overall network performance, just hides the badness
[02:28] <bob> ifu turn off logging then u don't know what people are getting
[02:28] <[VIP]Jamgoggâ„¢> if your sharing files why would you care what their getting?
[02:28] <bob> and using segments just encourages bad behavior
[02:28] <[VIP]Zaszâ„¢> i don't need to worry about what people are getting, as i only share what i want people to get
[02:29] <[VIP]Zaszâ„¢> what kind of bad behavior?
[02:29] <bob> because i want to know what is popular and who got what
[02:29] <bob> when ur downloading in segments, it just increases the number of transfers taking place, and hides the bad effects of people doing too many transfers
[02:30] <[VIP]Jamgoggâ„¢> you mean using up your bandwith
[02:30] <bob> it's better to have single fast transfers and limit their number, but people are dumb/greedy so they don't do that
[02:30] <bob> yeah
[02:31] <[VIP]Zaszâ„¢> dc does that with me whenever i run it anyway
[02:31] <[VIP]Jamgoggâ„¢> dont limiting slots acheive that
[02:31] <bob> every connection has an overhead that eats into total bandwidth, the more transfers, the less efficient ur overall transfer is
[02:31] <bob> only if u limit slots and max number of dl's and max speed to cut off new dl's
[02:32] <[VIP]Jamgoggâ„¢> are slots even a factor in strong
[02:32] <[VIP]Zaszâ„¢> i have a 10\10, and one file per user limiting
[02:32] <[VIP]Zaszâ„¢> it works for me
[02:32] <bob> of course it "works" but so does riding a moped to work
[02:33] <[VIP]Jamgoggâ„¢> I assumed that there was a way to limit how many users could connect to a file
[02:33] <bob> in fact thats a good analogy, what's better, riding a whole bunch of mopeds to work, or riding in one van on the highway
[02:34] <[VIP]Zaszâ„¢> i don't really mind the inneffective dl-speed, it's still faster than my old fuldc
[02:35] <bob> oh u don't mind getting crap speeds :D
[02:35] <[VIP]Jamgoggâ„¢> so strong doesn't help your download speed
[02:35] <[VIP]Zaszâ„¢> it helps me
[02:35] <[VIP]Jamgoggâ„¢> even though your getting segments from several users
[02:35] <bob> it can help *you* but it will tank the overall network
[02:35] <[VIP]Zaszâ„¢> before i never had speeds above 30 k
[02:36] <[VIP]Zaszâ„¢> now i've hade 300
[02:36] <[VIP]Jamgoggâ„¢> so the hub suffers
[02:36] <[VIP]Zaszâ„¢> ask loki about that
[02:36] <[VIP]Zaszâ„¢> he's the one who reccomended the use to me
[02:36] <[VIP]Jamgoggâ„¢> yea, I noticed he uses strong
[02:36] <bob> if it were up to me all segmenters would be banned
[02:36] <[VIP]Zaszâ„¢> good thing it's not up to you then
[02:37] <[VIP]Jamgoggâ„¢> heh
[02:37] <[VIP]Jamgoggâ„¢> hehe
[02:37] <bob> whatever
[02:37] <bob> it's a pyramid scheme for transfers
[02:37] <bob> if everybody used it, man, dc would die
[02:37] <[VIP]Zaszâ„¢> do i care, not really
[02:38] <[VIP]Jamgoggâ„¢> bob, whats better receiving 5 dollars a day from 1 person or 5 dollars a day from a hundred? which one makes you rich quicker
[02:38] <bob> btw networking/programming is my job so i'm not talking out of my ass
[02:38] <bob> thats not a good analogy
[02:38] <bob> 5 dollars a day from 5 people vs 25 dollars a day from one person
[02:38] <bob> which is more work
[02:39] <bob> getting the 25 dollars in one shot of course
[02:40] <[VIP]Zaszâ„¢> depends, is that a one time thing or a repeating thing?
[02:40] <[VIP]Jamgoggâ„¢> but does that one person have 25 dollars? like does one person have all the bandwith
[02:40] <bob> daily
[02:40] <bob> yeah
[02:40] <[VIP]Zaszâ„¢> not always
[02:40] <bob> if people did their transfers properly
[02:40] <[VIP]Jamgoggâ„¢> so then if the people you download from have the bandwith its better
[02:41] <bob> the problem is people dl'ing off u at 1.5 kB/s
[02:41] <[VIP]Zaszâ„¢> i try to disconnect my dl's that are that slow
[02:41] <[VIP]Jamgoggâ„¢> I do the same
[02:41] <bob> those people have too many transfers going, which slows down each transfer so it takes forever for anything to get done
[02:41] <bob> i think thats going to be a feature in the next dc++
[02:42] <[VIP]Zaszâ„¢> but i have to go offline now, talk to y'all latter
[02:42] <bob> do the right thing, get rid of the segmenter ;)
[02:42] <[VIP]Zaszâ„¢> that's why i limited my slots to 10\10
[02:42] <[VIP]Zaszâ„¢> never going to happen
[02:42] <bob> "not my problem" >:)
[02:42] <[VIP]Jamgoggâ„¢> so in theory limiting the amount of segments a person can get would solve the problem
[02:42] <bob> oh but it will be
[02:43] <bob> limiting to 1 segment, limiting max number of dl's, limiting max dl speed for a new dl to start
[02:44] <bob> the setting "no new downloads if download speed exceeds xxx"
[02:44] <bob> set that to 75-85% of ur total dl bandwidth
[02:45] <[VIP]Jamgoggâ„¢> limiting to one segement? That would be the same as dc++ 1 download 1 person
[02:45] <bob> exactly
[02:45] <bob> limiting ur own downloads is what makes that system work better than segmenting
[02:45] <[VIP]Jamgoggâ„¢> what about torrents they are faster than dc++ the more seeders the faster the speed
[02:46] <bob> only for the same reason that segmenting "looks" faster
[02:47] <bob> the only exeption is if ur own bandwidth vastly exceeds that of the people ur dl'ing from
[02:47] <bob> if u have a 10 mbit link and everyone else has 1 mbit links, u could have 5 dl segments and get 5 mbit/s
[02:48] <[VIP]Jamgoggâ„¢> I understand your point of view It makes sense. actually never tried strong or any others
[02:48] <bob> but that is rare, most connects are similar
[02:48] <[VIP]Jamgoggâ„¢> will probably stick with plain ol' dc++
[02:48] <bob> well, i work with one of the largest private networks in the world, so trust me on this :)


LA
Anata ga baka da! Repent!

Todi
Forum Moderator
Posts: 699
Joined: 2003-03-04 12:16
Contact:

Post by Todi » 2006-01-28 19:19

Lots of speculation.. but your deductions seem kinda shaky.

vellu
Posts: 20
Joined: 2003-08-30 01:33
Contact:

Post by vellu » 2006-01-29 04:03

I haven't tried segmenting DC myself, but I don't have anything against other people using them.

Don't know about the "overhead" thingy (many connections equals lot of overhead), but I would think that would be overshadowed by the fact that many slower sources add up to much more speed overall (usually) then one average/slightly above average source. Atleast that's what happens with Torrent, I don't see a reason why that wouldn't be true for DC segmenting aswell. One high speed source can be faster overall, no doubt, but those are super-rare too.

One negative aspect I can think of though. Segmenting user takes up way more downloads slots from other then a regular user, which could lead up (if lots people use segmenting) to much more frequent "no slots" from people. On the other segmenting also releases slots quicker since it gets smaller chunks of files.

The log-file thing, you can't really be serious about that? First entry in my upload log (which is the biggest log file I have) is dated July 2003, and the file is only 12 MB. Who cares, that's nothing. It would take 40000 years to fill up my HDD(160GB) with this logging speed! (dc running pretty much 24/7)

repattila
Posts: 5
Joined: 2006-01-25 11:06

Post by repattila » 2006-01-29 06:13

I think bittorrent has already proved, that segmented download has much more advantage than disadvantage.
The only reason against that may stand is that one could take a lot of upload slots, but so they will be free much earlier, so i don't think this reason would be enough to make people not to use segmented donloading.

imb
Posts: 99
Joined: 2004-06-15 17:48
Location: England

Post by imb » 2006-01-29 06:33

Gotta say I do have a feeling that multi source would be for the worse if it was implemented into the main client and everyone started using it. Look at Kazaa, WinMX, eMule, all take ages/are impossible to get anything done. It makes logical sense too, if one file takes 10 users and 10 slots up, rather than one file taking up 1 user with 1 slot, there has to be less slots. The negative aspects of more slots being open will be offset by the reduced time per transfer I hear you cry, however what about the people who log off as soon as their downloads are complete? Might be just speculation, but I've a feeling multisourcing will erode what is left of the DC community, and result in a 'no slots available' gridlock. Also I'd guarantee you I can get any mp3 album done quicker file swarming on DC, rather than on your crappy multisource torrents.

Pothead
Posts: 223
Joined: 2005-01-15 06:55

Post by Pothead » 2006-01-29 10:08

There is normally 2 main arguments against multisource.
1. Spams hub with ConnectToMe's
2. Takes many many slots.

What a lot of people don't realise is that a DC++ user with many files in the download queue, each of which has many sources, will spam and take loads of slots, just the same as a multisource client.

Big Muscle
Posts: 72
Joined: 2004-01-23 14:45

Post by Big Muscle » 2006-01-29 10:22

Pothead wrote:1. Spams hub with ConnectToMe's
DC++ also sends a lot of ConnectToMe's until source with slot is connected.

StrongDC++ sends ConnectToMe's until max. 10 sources with slot is connected.

bastya_elvtars
Posts: 164
Joined: 2005-01-06 08:39
Location: HU
Contact:

Post by bastya_elvtars » 2006-01-29 11:01

imb wrote:Gotta say I do have a feeling that multi source would be for the worse if it was implemented into the main client and everyone started using it. Look at Kazaa, WinMX, eMule, all take ages/are impossible to get anything done.
Yes, like I was tryin' to get a file with DC++ and in 4 days it didn't even start, now with eMule I got it in a night. With BitTorrent, fantastic speeds can be achieved, with DC you can only dream of them. Wonder what this opinion is based on. :?:

Moreover, I don't clearly see what the difference is between having 20 files in the queue and fetching each from 1 peer, and having 4 in the queue, fetching each from 5 peers. Same amount of $CTMs sent and slots engaged. Just wanted to point out: hardly there are any valid arguments against multisource.
Hey you, / Don't help them to bury the light... / Don't give in / Without a fight. (Pink Floyd)

Pothead
Posts: 223
Joined: 2005-01-15 06:55

Post by Pothead » 2006-01-29 11:10

bastya_elvtars wrote: Moreover, I don't clearly see what the difference is between having 20 files in the queue and fetching each from 1 peer, and having 4 in the queue, fetching each from 5 peers. Same amount of $CTMs sent and slots engaged. Just wanted to point out: hardly there are any valid arguments against multisource.
That's what i was trying to say. :) hehe, you just said it better. :)

GargoyleMT
DC++ Contributor
Posts: 3212
Joined: 2003-01-07 21:46
Location: .pa.us

Post by GargoyleMT » 2006-01-30 12:06

imb wrote:Look at Kazaa, WinMX, eMule
I'm not sure you can deduce anything meaningful from such a comparision. In DC, you have a maximum of 10,000 users or so per hub you have open. I'm sure most regular users have far less. The system in DC can't easily be compared to those networks, which are more centralized and have many more users visible to a single client.

Dr.Masturbator
Posts: 5
Joined: 2006-01-18 23:28
Location: inmywalett
Contact:

Post by Dr.Masturbator » 2006-01-31 11:38

i've been using revconnect since 2003 and never had any problems with it .never got kicked from a hub for using it, never "filled up my hard drive with some sort of logs", and users or ops never complained about me taking too many slots.in addition few weeks ago my hard drive died and i had to reinstall windows and everything .in 3 weeks i downloaded 175 Gigabyte of stuff with a budget broadband connection, now i have to buy a another drive coz i need more space.....Dc ++ is gonna HAVE to implement multisourcing one day if its wanna stay on the top of the"p2p market" the question is when...
i have no interest

ivulfusbar
Posts: 506
Joined: 2003-01-03 07:33

Post by ivulfusbar » 2006-01-31 15:32

I must say this thread is completely dead. The arguments are lacking horrificly. Like: "[02:27] <bob> each chunk makes a log entry which fills up ur hard drive ". Don't confuse a bad implimentation with a feature. Most of the arguments discusses bad implimentation. Its horrific to try to make an argument linking bad implimentation with a theoretical feature.

Almost only thing i can agree with is stupidity and greediness of users. But thats about it. Since most users have assymetric downloads, and if they only want to download one file, its in their interest to download from as many sources as possible to achieve higher download-speed.

And about the CTM-flooding. Its as horrific as any other argument in this thread. Its a common misstake to think that CTM takes bandwidth. It would not. It could do in a crappy implimentation. Since CTM is a 1-1 message (a message from one client to another) the bandwidth used up will be neglictable. It will only matter if you have an order O(N) where N is number of users online in the hub as sources and you sent it to all those users. And i have never in my life seen a single file with O(N) sources. And if it had O(N) users, O(N) users will probably not want to have more of it. ;)) (side-comment)..

i-find-this-thread-horrificly-lacking-good-arguments-ly'ers.
i-laugh-at-the-idea-that-people-link-crappy-implimenation-to-theoretical-featurishis-logic-ly'ers.
Everyone is supposed to download from the hubs, - I don´t know why, but I never do anymore.

Pothead
Posts: 223
Joined: 2005-01-15 06:55

Post by Pothead » 2006-01-31 17:10

ivulfusbar wrote:And about the CTM-flooding. Its as horrific as any other argument in this thread. Its a common misstake to think that CTM takes bandwidth. It would not.
ConnectToMe's get passed through the hub. Many connecttome's from many users will have an effect on the hub's bandwidth, multisource or not.
ivulfusbar wrote: i-find-this-thread-horrificly-lacking-good-arguments-ly'ers.
i-laugh-at-the-idea-that-people-link-crappy-implimenation-to-theoretical-featurishis-logic-ly'ers.
well put. :D

ivulfusbar
Posts: 506
Joined: 2003-01-03 07:33

Post by ivulfusbar » 2006-01-31 18:55

Pothead wrote:
ivulfusbar wrote:And about the CTM-flooding. Its as horrific as any other argument in this thread. Its a common misstake to think that CTM takes bandwidth. It would not.
ConnectToMe's get passed through the hub. Many connecttome's from many users will have an effect on the hub's bandwidth, multisource or not.
Yes, but an irrelevent amount of bandwidth. Since its a 1-1 message and neglictable compared to auto-search unless you have O(N) possible sources in your queue.
Everyone is supposed to download from the hubs, - I don´t know why, but I never do anymore.

GargoyleMT
DC++ Contributor
Posts: 3212
Joined: 2003-01-07 21:46
Location: .pa.us

Post by GargoyleMT » 2006-02-01 11:54

Pothead wrote:ConnectToMe's get passed through the hub. Many connecttome's from many users will have an effect on the hub's bandwidth, multisource or not.
It's not that they have no effect, but that a single broadcast command is equal to (potentially thousands) of directed commands. Directed commands just have very little effect compared to broadcast commands. fusbar is a mathematician, think in terms of limits and bounding terms.

Feniel
Posts: 1
Joined: 2006-02-26 11:46

Post by Feniel » 2006-02-26 11:54

perhaps something like this feature that i'm requesting would help...

http://dcpp.net/bugzilla/show_bug.cgi?id=847

Carraya
Posts: 112
Joined: 2004-09-21 11:43

Post by Carraya » 2006-02-27 03:07

As you can see from the comments, it's pretty certain that it won't happen...
<random funny comment>

Locked