Spidering hubs
Moderator: Moderators
Spidering hubs
I want to create a program (C++ or VB can work good) that periodically checks the hubs for users, description and name.
This will be used to update a PublicList.
Any ideas about that? What does DC Hub protocol say about bots?
I don't want to always grab lists and paste them in a global database (what my site is doing now), I want to have REAL data on hubs (some lists are fake, as you may know).
Only Dreamland.gotdns.org and P2PItalia.com are using that kind of Spider-Bot, and they don't want to give others their codes. So I want to make new codes for that.
This will be used to update a PublicList.
Any ideas about that? What does DC Hub protocol say about bots?
I don't want to always grab lists and paste them in a global database (what my site is doing now), I want to have REAL data on hubs (some lists are fake, as you may know).
Only Dreamland.gotdns.org and P2PItalia.com are using that kind of Spider-Bot, and they don't want to give others their codes. So I want to make new codes for that.
DJ ECHELON
MASTER OF BIT TORRENT
WEBMASTER http://www.p2pmania.it
My FAST PublicHubList http://www.p2pmania.it/hublists/PublicH ... config.bz2
MASTER OF BIT TORRENT
WEBMASTER http://www.p2pmania.it
My FAST PublicHubList http://www.p2pmania.it/hublists/PublicH ... config.bz2
First let's take care of hubs that want to be in a public list, and then of this. By the way, if you don't tell anybody with a apublic list the address of your site, it won't be added!yakko wrote:kinda like robots.txt for websites. It's tough keeping a private hub private.
DJ ECHELON
MASTER OF BIT TORRENT
WEBMASTER http://www.p2pmania.it
My FAST PublicHubList http://www.p2pmania.it/hublists/PublicH ... config.bz2
MASTER OF BIT TORRENT
WEBMASTER http://www.p2pmania.it
My FAST PublicHubList http://www.p2pmania.it/hublists/PublicH ... config.bz2
???cologic wrote:djechelon: I would support explicitly supporting no robots-like protocol. Don't let hubowners hide behind false security.
You think hubowners should go to public list control panel and manually edit the number of their users?
The robot is the only way to get a correct value of users. I'm trying to Telnet some hubs to read their responses and send my commands.
I made a conclusion. This bot should be someway able to log in in the hub and retrieve user count. If hub is private, user count will be set to zero. The only way to fake user count is the mirroring of users, against which there is no way to defend.
By the way, I'm reading about hubs connecting to the hublist server. YHub itself has dreamland.gotdns.org integrated in the hublist field.
This way probabily hubs log in to a server like clients in order to be included in public list.
What can you tell me about it?
I've heard about hublist.org having a spider that scans each hub every 8 hours...
DJ ECHELON
MASTER OF BIT TORRENT
WEBMASTER http://www.p2pmania.it
My FAST PublicHubList http://www.p2pmania.it/hublists/PublicH ... config.bz2
MASTER OF BIT TORRENT
WEBMASTER http://www.p2pmania.it
My FAST PublicHubList http://www.p2pmania.it/hublists/PublicH ... config.bz2