Kademlia

[yolanjoy]

Hi,

I coouldn't find a reference to get Kademlia working with mldonkey. I've enabled it but I see this message

Kademlia is enabled | Kademlia Connectivity: NOT enough online peers ,there maybe a problem with incoming udp packets

I understand it needs boot peers and there is an option to put a list of IP addresses in servers.ini. I don't know where it get those.
Looking online , nodes.dat exist but I don't know how to use/load it in mldonkey.

Please advise,
Thanks

[Testerprivate]

The network is decentralized. Here is an example from my servers.ini, but IPs change all the time.
` Kademlia = {

(* List of IP addresses to use to boot Kademlia networks *)

boot_peers = [
("62.57.217.152", 1985);
("95.95.225.94", 22503);
("83.33.123.110", 14346);
("217.160.131.161", 15178);
("79.156.179.50", 4672);
("69.163.35.60", 4816);
("147.91.20.66", 18356);
("79.116.48.99", 16965);
("176.185.128.201", 5264);
("90.77.185.202", 60185);
("37.134.73.139", 4880);
("219.78.91.107", 24672);
("194.2.0.20", 53);
("88.5.45.153", 25020);
("90.20.210.98", 4672);
("189.100.246.23", 41441);
("94.177.167.104", 4953);
("94.34.119.100", 4511);
("89.67.166.67", 6693);
("95.21.51.65", 21464);
("212.233.42.166", 16519);
("86.153.20.223", 6772);
("81.36.96.175", 1493);
("46.117.117.7", 4672);
("84.77.152.221", 4672);
("98.128.144.15", 4670);
("88.26.86.24", 4321);
("99.232.32.218", 16903);
("81.36.194.115", 21185);
("91.163.0.14", 61390);
("89.186.145.226", 16897);
("85.215.165.180", 4672);
("220.125.49.62", 4803);
("93.42.74.82", 4672);
("80.103.210.13", 1590);
("78.198.183.112", 9561);
("135.23.188.133", 37136);
("80.129.159.146", 9672);
("91.117.192.160", 23138);
("185.74.223.229", 10932);
("88.21.169.160", 4672);
("79.8.134.164", 62296);
("88.18.52.244", 4672);
("2.138.150.107", 1828);
("84.77.66.63", 1400);
("191.190.199.209", 4672);
("213.195.164.128", 4677);
("95.23.24.216", 13926);
("37.120.167.200", 13451);
("88.12.9.189", 41556);
("88.12.251.245", 54616);
("88.14.254.237", 6969);
("85.127.56.38", 4672);
("79.116.156.3", 2291);
("82.53.14.178", 7981);
("159.196.88.17", 7810);
("93.55.93.148", 4672);
("37.134.232.102", 4672);
("188.78.86.160", 6999);
("85.57.209.70", 4672);
("77.27.92.1", 1961);
("188.127.166.68", 7346);
("37.59.62.216", 12156);
("79.255.7.183", 37742);
("81.56.13.170", 16394);
("62.99.9.185", 4535);
("81.35.105.66", 20040);
("86.235.26.69", 7121);
("193.84.70.82", 24197);
("88.6.57.97", 47949);
("79.144.96.173", 3333);
("95.250.69.133", 4237);
("173.243.42.52", 1178);
("1.36.15.121", 6709);
("78.112.172.240", 4672);
("85.84.66.24", 4672);
("192.234.79.143", 7230);
("87.15.177.6", 6419);
("159.196.89.240", 4127);
("90.153.25.173", 4672);
("90.77.130.171", 4670);
("90.73.217.130", 5742);
("59.10.44.158", 60205);
("62.37.140.235", 40002);
("93.209.52.155", 12226);
("114.35.54.30", 24115);
("37.134.140.147", 4672);
("87.223.147.85", 5070);
("152.171.143.182", 3612);
("88.17.204.207", 51281);
("213.152.161.35", 22792);
("88.24.184.190", 62575);
("104.244.78.5", 20849);
("185.121.35.49", 18715);
("89.154.205.30", 6306);
("219.248.117.24", 4713);
("83.38.100.23", 4511);
("190.220.163.92", 17971);
("85.193.204.171", 52218);
("88.172.200.225", 16334);
("104.160.44.228", 11031);
("185.50.119.172", 4671);
("93.34.147.242", 39207);
("81.32.118.211", 5060);
("175.213.87.189", 4803);
("91.115.77.49", 17966);
("88.26.150.130", 1985);
("87.5.206.20", 3737);
("68.100.212.168", 57717);
("83.51.140.201", 62110);
("79.200.153.208", 4672);
("81.33.126.84", 4672);
("211.200.65.244", 4803);
("62.57.241.133", 4672);
("83.39.156.47", 50000);
("79.151.138.140", 60287);
("83.42.115.83", 1985);
("79.117.29.174", 5128);
("91.117.236.11", 52109);
("81.33.228.194", 61117);
("134.249.120.23", 17011);
("200.103.175.100", 34663);
("85.54.140.253", 14927);
("93.176.161.183", 1985);
("82.84.17.183", 16626);
("93.43.222.52", 54496);
("81.217.14.69", 52446);
("37.11.100.238", 62482);
("88.6.126.87", 51567);
("2.37.63.48", 4672);
("77.26.5.87", 17275);
("81.169.224.76", 3725);
("94.73.36.171", 33964);
("89.130.231.70", 54520);
("46.6.8.71", 44672);
("79.158.149.49", 4676);
("79.24.105.235", 17597);
("79.159.227.205", 45982);
("50.127.112.178", 53682);
("186.158.200.84", 12885);
("174.102.65.145", 4672);
("207.188.180.192", 4672);
("79.154.47.191", 20278);
("37.134.58.51", 12307);
("87.216.65.82", 7346);
("85.137.151.68", 5886);
("2.239.16.85", 39878);
("82.58.88.73", 4672);
("87.13.230.13", 7040);
("82.60.148.151", 4672);
("93.46.119.24", 4672);
("211.218.157.56", 4672);
("91.116.168.125", 5556);
("109.227.152.26", 18880);
("84.75.67.109", 59388);
("83.43.153.23", 1485);
("213.194.152.92", 15155);
("121.129.204.27", 4803);
("87.20.12.206", 37094);
("90.25.9.5", 4672);
("81.43.187.253", 4672);
("188.26.198.147", 4672);
("95.19.211.59", 27540);
("188.134.72.54", 21809);
("46.136.127.238", 1985);
("31.150.248.253", 4672);
("2.138.150.107", 41550);
("79.116.0.185", 44044);
("80.137.78.206", 28268);
("93.149.40.130", 20241);
("88.218.11.2", 1985);
("81.109.121.90", 28886);
("2.140.75.109", 62111);
("109.227.148.164", 4572);
("37.35.162.237", 4672);
("79.9.138.156", 63370);
("79.151.139.181", 46631);
("83.200.109.102", 8330);
("82.58.160.206", 6419)]
}`
Of course, you should open ports, even if you are behind a NAT provider.

[yolanjoy]

Thank you for providing the list.

I suppose I only have to open Kad UDP port and not individual ports of each IPs above. Am I right?
Do you have to update these IPs manually whenever there is a change?
Where should I look for getting these IPs? Is there any URL for the list?
When I loaded the nodes.dat it gave quite a few overnet peers. I don't think these are the kad boot peers but overnet which is inactive?

Thanks

[ogarcia]

I have the feeling that the Kad network is not working as it should. In my particular case it appears to be operational, with enough peers and a good list of boot servers and yet it doesn't seem to find anything on that network.

I have made the test of installing an eMule in a virtual machine, put to download a file that in MLDonkey appears to me without sources and in eMule at the moment I found someone in the Kad network that had it. Then I made the verification of downloading a file from myself and in eMule my MLDonkey client appears as not connected to the Kad network.

And as I said before, in the statistics section -> Kademilia it says:
Kademlia is enabled | Kademlia Connectivity: Got enough online peers

[yolanjoy]

For me, MLDonkey shows this message even when I used the IPs given above.

Kademlia is enabled | Kademlia Connectivity: NOT enough online peers ,there maybe a problem with incoming udp packets

Maybe MLDonkey is not setup correctly in my case. Does it work for anyone at all?

@ygrek can you please confirm if Kad client in MLDonkey does work ? If it does, I will appreciate if you could provide some guidelines to set it up properly. Thanks

@ogarcia I wonder if you have tried to investigate the issue by enabling debug ? At least in your case Kad client can connect to the peers. Can you share your config ? I will investigate it as well. Thanks

[HinTak]

FWIW, did you just read the messge? "there maybe a problem with incoming udp packets". Are you getting any incoming UDP packets? Firewall / NAT settings etc.

[ogarcia]

@yolanjoy The truth is that there is nothing strange about my configuration:

 Kademlia = {

	(* Set this option to false if you don't want to receive new Overnet peers *)
   update_nodes = true

	(* The MD4 of the Overnet client *)
   md4 = "4608B2DBF75A72383EB74BB6BCC20B7B"

	(* Interval (in seconds) before republish files *)
   republish = 10800.

	(* allow extended search to search on overnet *)
   search_keyword = true

	(* Size of the filename storage used to answer queries *)
   store_size = 2000

	(* port for Kademlia *)
   port = 14868
}

The port is open in the firewall and the NAT rule is correctly configured in the router.

Full donkey.ini

 port = 14860
 max_connected_servers = 3
 reliable_sources = true
 ban_identity_thieves = true
 server_black_list = []
 force_high_id = false
 force_client_high_id = false
 update_server_list_server = false
 update_server_list_server_met = true
 update_server_list_client = false
 max_sources_per_file = 5000
 client_md4 = 4608B2DBF75A72383EB74BB6BCC20B7B
 client_private_key = "a key"
 enable_sui = true
 min_users_on_server = 0
 max_published_files = 200
 login = ""
 old_files = []
 Overnet = {
   protocol_connectreply_version = 44
   protocol_connect_version = 1044
   update_nodes = true
   md4 = "4608B2DBF75A72383EB74BB6BCC20B7B"
   republish = 10800.
   search_keyword = true
   store_size = 2000
   port = 12540
}
 Kademlia = {
   update_nodes = true
   md4 = "4608B2DBF75A72383EB74BB6BCC20B7B"
   republish = 10800.
   search_keyword = true
   store_size = 2000
   port = 14868
}
 max_xs_packets = 30
 check_client_connections_delay = 180.
 client_timeout = 40.
 keep_best_server = true
 connect_only_preferred_server = false
 max_walker_servers = 1
 walker_server_lifetime = 300
 log_clients_on_console = false
 propagate_sources = false
 keep_sources = true
 black_list = true
 port_black_list = []
 queued_timeout = 1800.
 upload_timeout = 600.
 upload_lifetime = 90
 upload_full_chunks = true
 upload_complete_chunks = false
 dynamic_upload_lifetime = false
 dynamic_upload_threshold = 10
 connected_server_timeout = 1800.
 upload_power = 5
 max_server_age = 2
 remove_old_servers_delay = 900.
 min_left_servers = 20
 servers_walking_period = 6
 keep_cancelled_in_old_files = false
 keep_downloaded_in_old_files = false
 send_warning_messages = false
 ban_queue_jumpers = true
 use_server_ip = true
 ban_period = 1
 good_client_rank = 500
 options_version = 4

NOTE: All MD4s are invented (not real)

[HinTak]

"incoming udp packets". You need to set up port forwarding at the router if you are behind NAT. Many consumer grade routers don't even give you the option to set up port forwarding at NAT'ed routers. (It is more a hosting feature for hosting for virtual server farms, which many consumer network explicitly disallows in their service agreement) . If you don't comprehend any of this, you probably have not set up your router correctly, or your router is of limited functionality and does not provide this option. Also, your ISP might block "incoming" UDP packets of a certain kind at their discretion. (To test that you'll basically need to move your computer to a different network...). Anyway, too much already.

[yolanjoy]

I've port forwarding setup on the router. I Installed eMule v070a and have kad is working within it.
Having both running and testing out "search"on both. I get a feel that emule network is kinda flaky... probably same situation with Kad network I guess.

I think Mldonkey is much more flexible and have more features but many of them have not been documented ( at least I couldn't find ) .

[RigacciOrg]

I just installed mldonkey 3.2.1 compiled from sources, it seems that I cannot get responses from the Kademlia network. If I execute the kad_stats I get the Kademlia Connectivity: NOT enough online peers ,there maybe a problem with incoming udp packets response.

I executed a kad_load nodes.dat using a nodes.dat file copied from a well running aMule installation: I can see the outgoing UDP packets, but none of the hosts respond to me (the mldonkey host is a GNU/Linux machine with public IP address and the UDP port is open).

I also tried a kad_boot [ip] [port] using the IP address and port of the running aMule host; I can see the UDP packet arriving to the aMule host, but aMule does not respond either.

May be the outgoing UDP packets are not well formed?

[0xfk0]

I can confirm, that Kadmlia doesn't work. Neither if nodes.dat file provided, nor if kad_boot command is given manually. MLNet 3.2.1 compiled from github sources.

[0xfk0]

I have found, that no network activity (no outgoing packet to given ip/port) after givin kad_boot command. So, this is not a firewall problem.

[0xfk0]

I am not right. After enabling kademlia and restarting it tries to communicate with remote hosts, after giving command kad_boot ip:port:

14:24:37.285934 IP MY_IP.19532 > OTHER_IP.7346: UDP, length 27
14:24:37.369799 IP OTHER_IP.7346 > MY_IP.19532: UDP, length 529
14:24:44.324364 IP MY_IP.19532 > OTHER_IP.7346: UDP, length 27
14:24:44.394459 IP OTHER_IP.7346 > MY_IP.19532: UDP, length 27

After this, communication with specified ip/port stopped.

Results of kad_boots output now is the folowing:

87.216.x.x:73xx
115.234.x.x:63xx
87.27.x.x:46xx

After this, mlnet tries to communicate with a few hosts by sending 27 bytes, but receives no reply. And after some time kad_boots shows empty list.

[0xfk0]

After trying another IP:PORT pair with kad_boot command I got enough onlne peer!

[0xfk0]

Looks like Kademlia search doesn't work at all. It prints non-zero numbers in statistics, but search results completely depend on server responses only, as I think. How to test the search function?

[ygrek]

(ftr I never used edk network and have no idea how it works so cannot give much help here, if there are well-identified code bugs I will try to fix what I can)