BitTorrent

Technical discussion about the NMDC and <a href="http://dcpp.net/ADC.html">ADC</A> protocol. The NMDC protocol is documented in the <a href="http://dcpp.net/wiki/">Wiki</a>, so feel free to refer to it.

Moderator: Moderators

Locked
DynamiteNeon
Posts: 5
Joined: 2003-01-25 00:21

BitTorrent

Post by DynamiteNeon » 2003-01-25 00:36

Hi,

I was just thinking that DC might be an appropriate candidate to apply the bittorrent idea of swarmcasting.

http://bitconjurer.org/BitTorrent/index.html

I realize it might require a bit of an overhaul of the protocol, but I think that what bittorrent brings to the table is something to look into. Obviously, the main thing missing right now is file hashing, but it would also make you rethink the idea of slots since, in this case, you would benefit from more people sharing the same file than less.

I just thought DC might be more appropriate since it's more centralized around a hub than some of the other p2p apps out there.

I've been using the plugin for a little while now, and so far, it's been pretty impressive when large numbers of people are sharing the same file.

If you download the plugin, you could try your hand here to get a feel for how it works.

http://www.animesuki.com/

DynamiteNeon
Posts: 5
Joined: 2003-01-25 00:21

Post by DynamiteNeon » 2003-01-25 00:41

http://sourceforge.net/projects/bittorrent/

the link I posted there was down for some reason when I tried it, but here's the sourceforge link.

yakko
Posts: 258
Joined: 2003-01-27 01:04
Contact:

Post by yakko » 2003-01-27 01:10

I've had multiple people in my hub tell me I should switch to this in the past few weeks. Is this a new type of spam? My hub is a small community and we like it like that. From what I can tell Bitttorrent can't beat DC in that regard. Anyone else having people PMing ops about that?

DynamiteNeon
Posts: 5
Joined: 2003-01-25 00:21

Post by DynamiteNeon » 2003-01-29 19:14

You seem to be misunderstanding what I was suggesting. I'm not saying to get rid of hubs or ditch bittorrent. I'm saying to add the swarmcasting functionality to the protocol to improve bandwidth usage. The bandwidth could still only be shared among the various people on a particular hub. You wouldn't lose anything.

DynamiteNeon
Posts: 5
Joined: 2003-01-25 00:21

Post by DynamiteNeon » 2003-01-29 19:19

"You seem to be misunderstanding what I was suggesting. I'm not saying to get rid of hubs or ditch bittorrent."

typo. I meant you wouldn't have to ditch DC. BitTorrent is a separate application in and of itself. This was a suggesting to take a good idea from it and improve DC, not to ditch DC altogether.

A
Posts: 17
Joined: 2003-02-02 05:55

Post by A » 2003-02-03 09:57

i'm not an expert but isn't it true, that in order to do this
you must trust all the users in the hub?

Maby it is a better idea to spread it in 2,
1 server for search
1 for connecting

Shades
Posts: 2
Joined: 2003-02-26 15:04
Location: United Kingdom
Contact:

Post by Shades » 2003-02-26 15:19

Just watching how many users queue the same file from me, especially if it's relatively new shows how much a bittorrent like addition to dc++ would help speed up data transfer for all but the first people in the queue (who would remain unaffected).

Also the way bittorrent handles upload redeuces the annoyence of file leaches.

As both the bittorrent and dc++ source are opensourced i'm willing to have a go at combining them or help anyone else attempt it, the main problem i can see is the tracker which needs a static ip, although either a "plugin" type for hubs or using existing bittorrent trackers would solve this.

Anyway i'd really like a swamcasting feature to be added to dc++, i can see no drawbacks and lots of benifits for the community.
... and I was never here.

GargoyleMT
DC++ Contributor
Posts: 3212
Joined: 2003-01-07 21:46
Location: .pa.us

Post by GargoyleMT » 2003-02-26 16:12

Shades wrote:Anyway i'd really like a swamcasting feature to be added to dc++, i can see no drawbacks and lots of benifits for the community.
What I haven't seen here is a good explanation of the features that anyone would like to lift^H^H^H^Hborrow from BitTorrent. Would that be forcing sharing of currently downloading items, coupled with source exchange, and multi-source downloading?

Or would you literally like to have DC++ act as a seed for BitTorrent files? Or have DC++ be able to download .torrent files?

And no drawback and all benefits? Cool, you don't see that too often. :-D

sandos
Posts: 186
Joined: 2003-01-05 10:16
Contact:

Post by sandos » 2003-02-26 17:28

Shades wrote:Anyway i'd really like a swamcasting feature to be added to dc++, i can see no drawbacks and lots of benifits for the community.
Just wait for TTH + GetZBlock (or whatever the name was from Arne) + yet unnamed ability to share partial files and we would have this done easily. Dunno which way would be easier, probably the bittorrent way.

Shades
Posts: 2
Joined: 2003-02-26 15:04
Location: United Kingdom
Contact:

Post by Shades » 2003-02-26 20:49

i'm not an expert but isn't it true, that in order to do this
you must trust all the users in the hub?
as far as i know the bittorrent allows anyone to connect (or does so in my experience) so theres no level of trust at all in that, i assume in theory you could fake file blocks to mess up someones torrent, but in a system where upload doesn't effect your rating (read kazaa) theres no reason i can think of that someone would do this, further you could use hashcodes on each block or the file as a whole to limit this kind of behaviour.
What I haven't seen here is a good explanation of the features that anyone would like to lift
Pretty much as you mentioned would be the downloaders sharing the file chunks they have and so providing multiple sources, this would of course mean giving dc++ support for either .torrent files themselves or somthing similar. Of course if you did support .torrent files you'd have the added advantage of being about to easily code a bot that could "share" whatever a certain bittorrent tracker had availble. We would have to sort some method to alter the slot system to support this, either having an extra slot just for each torrent file or maybe you could only download from a downloader if they had a slot free.

To take advantage of the hubs there should also be some way to sharers of the same file can both intially combine a torrent (although i belevie the source includes this - my python coding skills are weak), or maybe a centeralised hub bot could be coded to act as a global tracker for that hub.
Or would you literally like to have DC++ act as a seed for BitTorrent files? Or have DC++ be able to download .torrent files?
theres not reason you couldn't have dc++ download .torrent files, or act as a seed themselves but i don't think we need to go that far just to have swamcasting, although i guess technicaly every file you shared with dc++ would be a seed this way.
... and I was never here.

eHast
Posts: 18
Joined: 2003-01-09 02:36
Location: Lund, Sweden

Post by eHast » 2003-03-01 16:06

There's also an older program called SwarmCast which does the same thing as BitTorrent (but better). It never got very popular unfortunately.

And the idea to include in DC has been around for a long time. But IMHO the protocol is basically too broken to support such big changes.

GargoyleMT
DC++ Contributor
Posts: 3212
Joined: 2003-01-07 21:46
Location: .pa.us

Post by GargoyleMT » 2003-03-03 20:38

eHast wrote:And the idea to include in DC has been around for a long time. But IMHO the protocol is basically too broken to support such big changes.
I think it'd be a mistake to try to hack the code in. Just figure out what the best lessons are and figure out how to most elegantly graft the same features into DC++. Hashes are doable, as are multi-source downloads. Sharing incompletes is as well.

But DC is centralized around hubs, and bittorrent was designed for a different purpose.

It seems that some of this is just idle talk anyhow, once we get code or a real proposal... wheeee!

Flash81
Posts: 1
Joined: 2003-03-04 09:46

Post by Flash81 » 2003-03-04 10:39

Although I dont know exactly what the swarmcasting is, I have seen a sketch of the BitTorrents functionality. And to be honest I cannot see that such function would be of any help in DC++.
The reason is that there are not that many transfers that are getting the same files from the same user at the same time. The whole idea with bittorrent is to distribute the same content to many users. That is not the case with DC++. And also, there are already some kind of such functionality in DC++, although not immediately obvious. As soon as a user have downloaded 1 file of a new set of files, and if it also is downloaded to a shared directory, and the filelist is refreshed, and the receiving parties are searching for alternates, then the load is spread out.
A lot of ifs, but its more a matter of time really. The file will eventually get shared, and eventually, the clients will search for alternate locations.

HaArD
Posts: 147
Joined: 2003-01-04 02:20
Location: Canada http://hub-link.sf.net
Contact:

Post by HaArD » 2003-03-05 08:14

True enough... all you really need to add to DC is File Hashes and multi segmented downloads and you've got the best of both really....

HaArD

sandos
Posts: 186
Joined: 2003-01-05 10:16
Contact:

Post by sandos » 2003-03-05 08:25

HaArD wrote:True enough... all you really need to add to DC is File Hashes and multi segmented downloads and you've got the best of both really....

HaArD
Well, minus the (possibly) much larger number of sources in bittorrent. Oh well, yes you can connect to multiple hubs and maybe get alot of sources in DC aswell, but the drawback is that the network isnt connected (between hubs).

eHast
Posts: 18
Joined: 2003-01-09 02:36
Location: Lund, Sweden

Post by eHast » 2003-03-08 16:35

Flash81 wrote:Although I dont know exactly what the swarmcasting is, I have seen a sketch of the BitTorrents functionality. And to be honest I cannot see that such function would be of any help in DC++.
The reason is that there are not that many transfers that are getting the same files from the same user at the same time. (...)
Yes this is more or less what I meant in my previous post.

There is no problem with putting multi-source downloading into DC++ now. This would be quite trivial and it's actually implemented in some DC clients.

The problem is that the current system is not designed for it. And although it pretty much sucks now that's nothing compared to how it would be if these features were used by everyone.

The main problem is that what you /really/ want from an application like this is to get the file as fast as possible. For the network you want to try to keep the total speed as high as possible. DC has no real way to do this. Even "trivial" things like putting highest priority on local users has no support in the protocol.

The only way to have a "working" multi-source downloading with hub software as I see it is to scrap the entire "slot" idea and limit based on bandwidth instead.

xl_kain_lx
Posts: 3
Joined: 2003-05-06 00:10

my two cents

Post by xl_kain_lx » 2003-05-06 00:28

Hey guys, this is my first post here... i looked around to see if someone brought this up before i would.

i have to say though that this idea really makes alot of sense in the long run. incorporating the overall concept of the bittorrent technology into DC++ would make it easier for the distribution of everything available on DC. besides that, making DC++ compatible with .torrent files as that other guy described would really make DC++ a much more dynamic P2P program. DC and bittorrent users alike would benifit greatly at the same time. i'm sure if the developers contacted whatever that guys name is about figuring out a way to do this, would find it to bo difficult, but it really would be a milestone in DC++ development.

Utreife
Posts: 1
Joined: 2003-05-08 05:55

Post by Utreife » 2003-05-09 15:47

Probably I am using a forbidden name here, but eDonkey/eMule are using this Swarmcasting-technique as well; multi-source downloading, automatic incomplete sharing of downloading files, linked upload/download limit (but if upstream >= 10: download = infinite;). Fast searching of all served files through the whole network is another feature. Oh well, I am sure you can find the information if it sounds interesting.

What those above mentioned clients are missing, is the diversity of files and the way the community is divided over servers. On the other hand, DC++ is missing a global search of all hubs. Yes, there is MoGlo, but it's not widely accepted and is, with the forementioned being one reason, certainly not reliable for getting the complete information.

Where the eDonkey client is linked to a general Overnet network and a big server containing a list of all served files in the network, DC++ clients connect to a number of hubs. This is as well one of the great strengths of DC++. It improves efficiency on several levels. People go where they need to go and the files are there where need to be. Moreover, the power over the hubs lies in the hand of (some of) the users, not with someone owning a huge server supernode kind of thing with a huge pipe supporting it.

So if there is a way to implement Swarmcasting-technique, since a global search facility is not easily achieved, it should be on hub level. See, I doubt the truth of something Flash81 said:
[quote=Flash81]The reason is that there are not that many transfers that are getting the same files from the same user at the same time.[/quote]
In specific hubs, for instance anime hubs, a lot of people who are in there are looking for the same files, but at the same time many users are having the looked for files in common. I believe efficiency can actually get a lot higher if Swarmcasting were used on this occasion.

You might wonder why I take all this text to point out the obvious, but you may understand in my conclusion, which reads:

..::CONCLUSION::..

(lol)

Anyway... while you might be developing global Swarmcasting such as Bittorrent is using, and you notice it hardly can be done, remember that the way DC++ users are sharing needs no more than hub-level Swarmcasting. Which could prove easier to develop. Maybe a more important result is that DC++ will stand out as the best of both worlds. It will not be just another Swarmcasting clone, but will use resources more efficiently while having a less straining community, with user controlled hubs.

Also, a balanced linked up/downstream limit combined with Swarmcasting could mean people don't have to fakeshare anymore; traffic is more homogenic, more equally divided. Users with ultrafast connections/huge bandwidth who share below their capacity would be renderd less idly as well. Though, these are humble speculations. Maybe it's just interesting to see how it works in reality, on small scale for instance in some stereotypical hubs and to predict the usefulness.

Yes, I'm done now.

volkris
Posts: 121
Joined: 2003-02-02 18:07
Contact:

Post by volkris » 2003-07-14 15:10

All putting bittorrent-type support into DC++ means is implementing the multi-source downloading that we've already been talking about.

That is all.

HaArD
Posts: 147
Joined: 2003-01-04 02:20
Location: Canada http://hub-link.sf.net
Contact:

Post by HaArD » 2003-07-14 19:51

sandos wrote: Well, minus the (possibly) much larger number of sources in bittorrent. Oh well, yes you can connect to multiple hubs and maybe get alot of sources in DC aswell, but the drawback is that the network isnt connected (between hubs).
Hmm... must be the hubs you hang out in.... :P

A lot of the most popular/recent downloads (especially when shared as the original RAR release) are available from multiple sources on DC. If people share their completed files, the number of sources grows as people download.

Maximizing D/L speed by allowing multiple sources to simultaneously feed a single D/L allows files to complete faster and become available for others to D/L.

Unlike Bittorrent, DC has more "stickiness" and people are far more likely to stay connected/share the files they have recently downloaded.
eHast wrote:There is no problem with putting multi-source downloading into DC++ now. This would be quite trivial and it's actually implemented in some DC clients.
As far as other clients that allow mutlisource downloads, the only Windows one I've heard about is pDC which by every review is buggy/unstable/unreliable. Please enlighten me as to the other options available for Windows.

Since DC does support RESUME it should be easy enough to implement along the same lines as FlashGet and other HTTP Download Managers. There must be more to it. If this was "trivial" or "easy" I would have suspected that lots of clients would have enabled it successfully.

HaArD

ender
Posts: 224
Joined: 2003-01-03 17:47

Post by ender » 2003-07-15 08:26

HaArD wrote:As far as other clients that allow mutlisource downloads, the only Windows one I've heard about is pDC which by every review is buggy/unstable/unreliable. Please enlighten me as to the other options available for Windows.
DCGui-Qt - if you can live with Qt interface and 30 day trial (of Qt)

HaArD
Posts: 147
Joined: 2003-01-04 02:20
Location: Canada http://hub-link.sf.net
Contact:

Post by HaArD » 2003-07-15 09:05

ender wrote:
HaArD wrote:As far as other clients that allow mutlisource downloads, the only Windows one I've heard about is pDC which by every review is buggy/unstable/unreliable. Please enlighten me as to the other options available for Windows.
DCGui-Qt - if you can live with Qt interface and 30 day trial (of Qt)
direct connect 4 linux - homepage

Not quite what I was looking for..... ;)

HaArD

ender
Posts: 224
Joined: 2003-01-03 17:47

Post by ender » 2003-07-15 09:22

look in downloads, there's windows version available

HaArD
Posts: 147
Joined: 2003-01-04 02:20
Location: Canada http://hub-link.sf.net
Contact:

Post by HaArD » 2003-07-15 14:10

Well, I'll be damned... Thanks ender :)

sandos
Posts: 186
Joined: 2003-01-05 10:16
Contact:

Post by sandos » 2003-11-04 19:41

sandos wrote:
HaArD wrote:True enough... all you really need to add to DC is File Hashes and multi segmented downloads and you've got the best of both really....

HaArD
Well, minus the (possibly) much larger number of sources in bittorrent. Oh well, yes you can connect to multiple hubs and maybe get alot of sources in DC aswell, but the drawback is that the network isnt connected (between hubs).
One idea to solve this is to have a hub dedicated to connecting people with similar interests. The main difference between DC (and most any p2p) and BitTorrent is that a BitTorrent client will see anyone interested in the same file as you, or those already having it. This is good because of the efficiency (no need to search for anything, we know everybody wants the file) and the large number of sources gathered (Ideally theres is only ever one torrent for each file, so as not to spread sources).

How could this be emulated in DC? Quite easily by creating a dedicated search-hub which ditches the traditional DC broadcasting of stuff. Instead the hub would silently collect data about what people want, and then somehow group those together and let them access eachother. Hashes would also make this alot more effective, but PFS (Partial File Sharing) would make it extremely much more effetive, sadly PFS is nowhere to be seen in DC. PFS requires hashing and specifically, on-the-fly checking of downloaded files to be of any use.

Yes, this goes against the only-share-to-those-in-the-same-hub principle of DC, but the good thing is if we dont want this, just dont join one of these hubs. Also, some logic in the client as to reward those that upload to us would be beneficial for this (its also in BitTorrent).

What problems would searchhubs solve? They should be able to easily hold 10k-100k users. Bandwidth will not be a problem, cpu might though. There are multiple ways the hub might match interests, and they all use some cpu.

One interesting aspect of these hubs is that they could group users which, apparently, hasnt got any interest in eachother, but whom satisfy eachothers needs in a more elaborate way than "I download from you, you download from me". The simplest might be a ring, A downloads from B, who downloads from C, who downloads from A.

Locked