ADC inform, an idea

Technical discussion about the NMDC and <a href="http://dcpp.net/ADC.html">ADC</A> protocol. The NMDC protocol is documented in the <a href="http://dcpp.net/wiki/">Wiki</a>, so feel free to refer to it.

Moderator: Moderators

Locked
lordadmira
Posts: 52
Joined: 2003-03-12 00:06
Location: Zinzinnati

ADC inform, an idea

Post by lordadmira » 2006-04-08 14:30

I had an idea about easing the hub load on new connections with ADC. Right now the hub sends the entire user list with all their information to the newly connecting client, right. A way to cut down the hub traffic is to have the hub instead inform the existing clients about the newly connected client, and have them tell the new client about themselves. That way the user info list is a client<->client communication rather than a hub<->client. So for example client A connects to the hub, with all its information, and a 16 bit random string to identify the login. The hub then sends a message to the existing clients, an "inform request", containing the new client's IP, port, and the 16 bit string that A originally sent, telling them to send all their information to client A. Client A will also give out all its information in this exchange. Client A knows which hub the replies go with based on the 16 bit string. This way the hub has the full client list and all the clients have the full list, but with a minimum of hub traffic. For passive clients, a reverse form of this can be used. For passive<->passive clients, the hub itself will have to give out the information.

Just an idea.

LA
Anata ga baka da! Repent!

TheParanoidOne
Forum Moderator
Posts: 1420
Joined: 2003-04-22 14:37

Post by TheParanoidOne » 2006-04-08 14:51

It seems to me that the amount of traffic has not changed. Just been rearranged. You changed it from "Give information about n users to 1 user" to "Give information about 1 user to n users". Have I interpreted your proposal correctly? If so, I'm not really seeing what load you are reducing.
The world is coming to an end. Please log off.

DC++ Guide | Words

lordadmira
Posts: 52
Joined: 2003-03-12 00:06
Location: Zinzinnati

Post by lordadmira » 2006-04-08 16:37

Ur right, the total traffic is the same but it has been rearranged for the benefit of the hub. In the current way, all the user list information flows over the hub's inet pipe to the new client. This way, only the inform request goes over the hub's inet pipe and the full load of the user list is distributed to all the users. If this were a relationship diagram it would be like:
old: 1:1
new: 1:n:1
Ur second paraphrase is wrong though, or atleast truncated. To carry on ur terminology it would be "Give information about 1 user to n users; n users give information about 1 user (self) to 1 user."
To go into a bytes perspective, the hub now transmits all user information, which can be of significant length. nick+share+connection+tag+email+mode etc. But if the hub is "I'm not telling this guy about u, u tell this guy about u" then it only has to send the length of ip+port+key. e.g. XXXXX123.123.123.123411Gp. Or whatever the ADC consistent formatting is. But u see that the bandwidth savings for the hub can be significant. Even greater if packed forms are used.

LA
Anata ga baka da! Repent!

ullner
Forum Moderator
Posts: 333
Joined: 2004-09-10 11:00
Contact:

Post by ullner » 2006-04-08 18:03

Excellent. HIs and OPs for everyone!

FarCry
Programmer
Posts: 34
Joined: 2003-05-01 10:49

Post by FarCry » 2006-04-08 18:27

That's certainly a good idea in general, but there are several issues in practice. On a large hub, a newly connecting user would suddenly get thousands of incoming connections at once from clients trying to get their info through. Firewalls won't like that and socket systems tend to break or not work correctly under such stress. The resulting raw traffic (including overhead for establishing the connection) would be much higher. If you imagine a hub reboot situation with thousands of clients connecting in a short period, the problem multiplies.

Reliability of connections is also a big problem. Not everybody can establish connections to everyone, even if they're not both passive. There would be no way for a client to notice that the user list is incomplete because some clients couldn't connect...

...or simply didn't. There's also the trust issue ullner hinted at. The hub doesn't see the information and thus can't verify it.

A more practical solution would be to use clients as "sub-hubs" to divide the broadcast responsibility, but that's not trivial either.

GargoyleMT
DC++ Contributor
Posts: 3212
Joined: 2003-01-07 21:46
Location: .pa.us

Re: ADC inform, an idea

Post by GargoyleMT » 2006-04-08 19:51

lordadmira wrote:I had an idea about easing the hub load on new connections with ADC.

It has its own problem: how can you trust such information? How does the hub know you're sending out the same information to users as you send to it? (Not a whole bunch of bad can come of this - the stuff in INF is relatively benign, except for what Ullner mentioned.

Based on the existence of "bad" clients and DC++ derivations to detect them (DCDM), I don't think this is a step in the right direction.

lordadmira
Posts: 52
Joined: 2003-03-12 00:06
Location: Zinzinnati

Post by lordadmira » 2006-04-08 20:41

FarCry wrote:a newly connecting user would suddenly get thousands of incoming connections at once from clients trying to get their info through. If you imagine a hub reboot situation with thousands of clients connecting in a short period, the problem multiplies.

Not necessarily. The hub can throttle the rate as necessary. The connect-transfer-disconnect only takes a fraction of a second anyway.

Not everybody can establish connections to everyone, even if they're not both passive. There would be no way for a client to notice that the user list is incomplete because some clients couldn't connect...

If two people can't connect to each other and they're non passive, then they're doomed from that point on. If I can't download from person X they're dead to me. I don't even need to know he exists, it doesn't do me any good.

...or simply didn't. There's also the trust issue ullner hinted at. The hub doesn't see the information and thus can't verify it.

We have that problem now. If there is a malicious client they can already ignore search requests or downloads. There's no way to control what a client does.

LA
Anata ga baka da! Repent!

lordadmira
Posts: 52
Joined: 2003-03-12 00:06
Location: Zinzinnati

Post by lordadmira » 2006-04-08 20:48

Obviously "I'm an Op" and similar privilege information has to come from the hub only. The client<->client info has to purely informative only.
Anata ga baka da! Repent!

ullner
Forum Moderator
Posts: 333
Joined: 2004-09-10 11:00
Contact:

Post by ullner » 2006-04-09 05:37

Yes. I'll just send 0 slots, IP 127.0.0.1, Port 0, Files in share 0, CID FOOMYBAR. If we break down what "informative" is, we're left with... what, e-mail and decription? I don't see the use of changing the protocol with that little gain for the hub.

GargoyleMT
DC++ Contributor
Posts: 3212
Joined: 2003-01-07 21:46
Location: .pa.us

Post by GargoyleMT » 2006-04-09 07:46

lordadmira wrote:We have that problem now. If there is a malicious client they can already ignore search requests or downloads. There's no way to control what a client does.

We have malicious clients now, and they do send out spurious information in MyINFO, but I think you can appreciate that the solution you've proposed adds another level to their ability to mislead people - it would be quite easy to broadcast one thing to operators and another to users. Sure, that can be detected eventually, but I think that makes the solution too unclean.

It might be wise to go back and look at the evolution of SCH, for although it was also proposed to be client to client, it's now advised to be used only in trusted environments. At least one of the concerns, flooding, is the same.

lordadmira
Posts: 52
Joined: 2003-03-12 00:06
Location: Zinzinnati

Post by lordadmira » 2006-04-09 20:26

"Informative" means anything not central to the operating of the hub system. It's just things "nice to know". U don't need to know any of MyInfo or even have the complete userlist to send search requests. IP and port is all u need for client<->client searches. Having a totally secure unspoofable network is against the laws of physics.

I guess I'm not averse to complexity for the sake of even moderate improvements.
Anata ga baka da! Repent!

ivulfusbar
Posts: 506
Joined: 2003-01-03 07:33

Post by ivulfusbar » 2006-04-10 00:11

But INF does not only contain informative message. It contains message you do need. Like: ipv4, ipv6 addresses, CIDs..
Everyone is supposed to download from the hubs, - I don´t know why, but I never do anymore.

FarCry
Programmer
Posts: 34
Joined: 2003-05-01 10:49

Post by FarCry » 2006-04-10 05:58

lordadmira wrote:Not necessarily. The hub can throttle the rate as necessary. The connect-transfer-disconnect only takes a fraction of a second anyway.

Establishing a connection can easily take much longer than a second. The hub also doesn't know how many connections per second a user's setup (firewall) will tolerate. You surely don't want the user list of a 10000 user hub to take an hour to build, just to be on the safe side.

lordadmira wrote:If two people can't connect to each other and they're non passive, then they're doomed from that point on. If I can't download from person X they're dead to me. I don't even need to know he exists, it doesn't do me any good.

He'll still get your main chat messages and you might even get search results from him. A connection may also work if it's established the other way around. The problem may also just be temporary and retrying for an unspecified time isn't a good solution. An operator or bot (I for one even as a normal user) would always want to see all online users, no matter if it's possible to get a connection to them.

footnmouth
Posts: 4
Joined: 2005-12-05 10:08

Post by footnmouth » 2006-04-11 10:06

too easy to attack the hub if you did it as proposed up top.
is there a site i can see the current white paper on ADC?
thanks


Locked