Hublist-list

aDe · Post by **aDe** » 2003-04-23 02:30

Just an idea I got due to all the fuzz about hublists.
We all know the concepts of clients, hubs, and the server that they register to. I'm suggesting a higher level in the "hierarchy", namely a hublist-list server. It could look something like this; on a standard xml on a http server (sourceforge, for instance)

Code: Select all

<Hublists lastupdated="2003-04-23">
   <Hublist url="http://ahublist.com/hublist.config" format="DC1">Hublist name here</Hublist>
   <Hublist url="http://ahublist.com/hublist.config" format="bz2">Hublist name here</Hublist>
</Hublists>

and so on. The lists could be displayed in a dropdown box in the dc++ hublists window. Good for users that don't know how to fiddle with first finding hublists and then setting them up in settings, and for the rest, too, MoGLO for instance =) and any other client interested in which hublists to use. Bandwidth you say? Hmm.. maybe. But the file will be rather small, not much bigger than version.xml, anyway.
The format tag?? Because DC++ only recognizes bz2 lists if there is a .bz2 extension. That means, trying to use querystrings on a php page won't work, unless you make a &x=.bz2 dummy variable come last. And it can be quite handy.
This page could be slimmed down too, I guess. Instead of <Hublist> just <HL> and so on. If the bandwidth is an issue that is.
Hmm, a value should also be included to set all clients to change to a different hublist-list, if the server is about to go to eternal sleep or if the bandwidth requirements can not be met. Or, several mirrors of the list could be hosted and the all the URLs included in dc++, but randomly select one when grabbing it to even out.
Well.. should be one thing or another to discuss here now. Hope you like my idea.

ivulfusbar · Post by **ivulfusbar** » 2003-04-23 02:50

*smiles*

A List of Lists, a matrix of possible hubs ;))

ivulfusbar · Post by **ivulfusbar** » 2003-04-23 04:03

well this would certainly help, or you code in a fixed lists of hublist-urls. And have people choosing from that one directly in the client. But then that would be restriction... hmm.. but you raise an important point. Would be alot easier if they could choose a hublist, and not have to look for a url on their own.

so-now-we-only-have-to-solve-best-way-of-getting-there-ly'ers ;))

Sedulus · Post by **Sedulus** » 2003-04-23 04:36

> Because DC++ only recognizes bz2 lists if there is a .bz2 extension.

since 0242 a "Content-Encoding: x-bzip2" header is also acknowledged

> List of lists

this would be really useful if you have a couple of content-specific hublists.
when it's just about taking load of servers, http302 redirecting can be used.

> Eternal sleep

yeah, if we had had that one for the current list........ good thinking :)

aDe · Post by **aDe** » 2003-04-23 05:50

Sedulus wrote:since 0242 a "Content-Encoding: x-bzip2" header is also acknowledged

oh, nice. thanks

Sedulus wrote:> List of lists

this would be really useful if you have a couple of content-specific hublists.
when it's just about taking load of servers, http302 redirecting can be used.

Yeah, that's the way I thought. There is already several content specific hublists.

Sedulus · Post by **Sedulus** » 2003-04-23 06:44

now... how should this list be configured in dc++?
- hublist, different extension (.list)
- hublist, let the parser decide whether it has fetched a HLL (hublistlist) xml or a regular hublist
- add an HLL URL configuration string (and perhaps remove the current public hublist box)

personally I'm not that fond of extensions, but parsing a bit to see what type a file is is even worse imo.

aDe · Post by **aDe** » 2003-04-23 06:55

> add an HLL URL configuration string (and perhaps remove the current public hublist box)

Yes, I'd go for that. Typically you shouldn't need to change this URL, unless there would be a server failure.

The HLL URL could be included in version.xml (if the server used is not sf)?

And then make a dropdown lists to pick your hublist in the hublistwindow. Some people will probably want to have a button that merges them all too, but I'm not sure if that's a good idea.

Moch · Post by **Moch** » 2003-04-23 13:39

Yeah, I think since the XML will already have format="blah" it doesn't matter what the extention is on the list, the client will know what to expect. There should just be some simple checking for web server errors (IE: 404).

Also... if anyone is worried about size of this xml file, why can't the file be bz2 and uncompressed on the fly. I am not sure how the bz2 decompression routine works that Arne is using, but the one I use in Linux allows you to decompress sections on the fly. Which means you dont have to keep very big chunks of data in memory.

Anyway, I think that this is a great idea. Perhaps someone could actually host the XML file and hublist owners would register there lists with them, that way people can update the xml file whenever they want to. I think that this needs to be done only by one person/site though, because if not we will start getting the same problem and need a hub list list list for the hub list list for the hub lists.. sheesh. I am not complaining at this level of hub list list abstraction though, because I think it is a great idea that there will be more than one hublist for new users to choose from so hopefully at least one of them would work. (Like you said).

Ahhh sometimes I feel like I should keep my fingers tied and just not post... *L*
~Moch

[ASH]Nev · Post by **[ASH]Nev** » 2003-04-24 00:22

I think this is an excellent idea

Post by **arnetheduck** » 2003-04-24 03:17

Moch wrote: I am not sure how the bz2 decompression routine works that Arne is using, but the one I use in Linux allows you to decompress sections on the fly. Which means you dont have to keep very big chunks of data in memory.

Which lib is that? To my knowledge, you have to decompress one block at a time more or less, and since libzip2 takes x * block size memory for both compression and decompression (x being the number I forgot), it not really streaming...that's why I decided for zlib instead of bz2 for the GetZBlock...in any case, such an xml would be what...200 bytes? 300?

Moch · Post by **Moch** » 2003-04-24 12:21

Perhaps that is my fault for assuming what a library is actually doing behind the scenes. I am using libbzip2 by Julian Seward (docs here)

It seems to be streamming to me. To use it you basically feed in the compressed data in increments of your choosing, and it spits out whatever it can uncompress so far from the data that you have given it. (Vice Versa for Compressing).

It could be just an illusion eh?

You are right though

Hopefully the XML file would only be a few hundread bytes. If it got past that then err there would be just too many hublists.

~Moch

ps. I still think the hublist list is a great idea

Post by **arnetheduck** » 2003-04-24 13:19

Yeah, but you have to feed it a lot of data before it starts spitting it out and that's the main problem with it...and the large memory requirement...and the slow speed...=)...but at least it compresses well...