If you are accessing this from the Web, not all links will work. The best way to get to this Flog is via Freenet.

Resources page

2009/10/27

Build 1238

1238 is now available. Normally I wouldn't blog on every build, but this one is worth talking about. The most important change is that this build has fixed a whole bunch of bugs in the client layer (persistent downloads and uploads), including stalling uploads and downloads, and it has done it in such a way that old, broken uploads should fail rather than just hang forever, old downloads should complete, and new uploads should complete. Some of this may have been a relatively recent bug according to the people I've been working with (notably p0s, although evanbd has helpfully nagged me about the problem), however there are definitely some longer term issues here too, so if you have stalling downloads please try 1238!

Another notable change in 1238 is the new CSS filter, kurmi's SoC project, has been merged. This is much more detailed and comprehensive than the old filter, and hopefully much more secure as a result: It parses everything and lets through only what it can understand, just like the HTML filter. This is the only way to be sure that a filter is secure. The old CSS filter only tokenised and didn't parse, so might (at least in principle) have let dangerous content through, and in order to prevent this it was more strict with strings etc than the new filter is. Also, the new filter supports > selectors, although not CSS3 selectors (which the old filter did, because it didn't parse them). It also has an extensive set of unit tests. Kurmi's mentor was nextgens and I finished it off, resolved parsing issues, added unit tests, and merged it. So it is a significant improvement, something that we would have had to implement sooner or later, and it might just be a security improvement.

Another change is that the first-time wizard is now considerably more concise, while hopefully being clearer and having less FUD. If you can do usability testing, please do - that is, find somebody who has never installed Freenet, ask them to install it, and note down wherever they get stuck, what questions they ask etc. Don't help them, at least not until after they have got stuck. If they get stuck, that is a usability bug. Please send the results to the devl list, or to me via Freetalk on the en.freenet board, so that we can solve these bugs.

Also, we have finally merged the new wininstaller, thanks to Zero3 and Juiceman, which does not create a custom user for Freenet, so should run into far fewer anti-virus/system policy/Vista compatibility issues, which were affecting very many users. Also, it includes a system tray icon from which you can conveniently start and stop Freenet if you need to for e.g. online gaming. So a bunch of important improvements, coming from various people. Worth a quick blog post IMHO.

Next up, the new Mac installer, which will also include a system tray icon, thanks to mrsteveman1. p0s is continuing to work on Freetalk, currently working on a login system (you might have to wait a day or two for me to catch up with and deploy Freetalk as the current version is incompatible with 1238); hopefully it will be ready to be official before Christmas.

Recent performance testing data shows that as long as you fetch immediately, Freenet can perform well, around 59 seconds to fetch a 1MB test key from a freshly bootstrapped node. Anecdotally very popular or very new keys can achieve very high speeds, maybe 30-80KB/sec. However, if you wait a few days, many of the keys will have fallen out, so it may fail and will probably be slower. Now, the long-term push/pull data (waiting e.g. 3 days between inserting and fetching) strongly suggests that some proportion of inserts are "getting lost" on the initial insert, pushed to the wrong part of the network due to backoff or some other reason. I am going to implement some test code to verify this theory; if it turns out to be correct, MHKs (duplicate the top block) and some tweaks to splitfile redundancy should be a fairly easy workaround, and improve persistence considerably.

We are also planning some more work on network level diagnostics. Currently, we have probe requests. Probe requests are a kind of request that does not return data, is routed to a location rather than a key, and returns information about the nodes along the way - their locations, UIDs (which are only used for probe requests) and whether they are backed off. This is conceivably somewhat useful to an attacker, although swap requests (which are a vital part of routing) reveal most of this information without probe requests. Probe requests help us to understand what is going on on the network, whether routing is working, whether churn is a problem, and how big the network is. But we are planning to get rid of probe requests in favour of more specific, more useful mechanisms:

• Random-routed requests returning the node's uptime and a unique ID for the node: This will enable us to do a quick estimate of the network size, similar to tag-and-release statistical sampling.
• Random-routed requests for actual data: These would be random routed for some number of hops and then turn into normal requests. This would allow us to test fetching a given key from a random node, so we can get a better picture of performance across the network as opposed to having to bootstrap a new node (for a bad result) or use a well established developer node (for a good result). This might also be useful to e.g. freesite authors, because you could determine whether given content is still available.
• Tracer requests: A special key type used only for testing, kept in a separate datastore. You could insert a test key, and then fetch it some time later or from a different node. Both on insert and request, the nodes' locations and UIDs will be returned.

Obviously these are temporary measures to help us to understand what is going on on the network and thus improve performance. One of the best ways to improve Freenet's security and survivability is to get a bigger network IMHO, and we will achieve this by improving performance. Probe requests suggest Freenet is around 2500 nodes at any given time, up slightly from a few months ago, while performance has improved.

Bloom filter sharing is still planned for 0.8, but may be postponed, it is more important to deal with the other network issues first as they may be easier to fix now. Most of the hostile environment / security stuff we have planned for 0.8 has already been implemented, there is more work to do on usability and integrated functionality, probably more work to be done on persistent downloads (e.g. backing up the database, less disk access for downloads), fixing XMLSpider, fixing the uninstall survey, etc. Sashee's web-pushing branch will be merged when it is ready, this might be after 0.8.

We have approximately 20 weeks left on current funding, this will be enough to get a reasonable 0.8 out, but there is much more we'd like to do. Hopefully performance and data persistence will be improved considerably in 0.8, and we may implement Bloom filter sharing, but there is usually more work to do on that, e.g. there is a second phase for Bloom filter sharing, and there are some plans for improving performance on slower links by reusing bytes from padding for transferring data. There is widespread support for a "pause mode", enabling nodes to be sent to sleep for a while while gaming but to be able to recover quickly once they are wanted again. Fully distributed, web of trust based searching is planned, and might eventually have a secure reinsert on demand mechanism. Passive requests would greatly improve chat apps and similar things, speeding them up, making them scale better, and reducing their network impact. Long-term requests would make low-uptime nodes and disconnected darknets much more useful. Sneakernet is obviously essential for some hostile environments, and the combination of sneakernet and long-term requests would give us something that could run anywhere you can swap USB sticks, or have phones do bulk transfer over wifi when you are close to your friends. Randomising the keys for inserted data would help considerably with security for large inserts, but there are issues with reinserts, so it might be a configurable option. Encrypted tunnels would improve security at the cost of quite a bit of performance, but combined with randomised keys we could just tunnel the top keys. Freemail should be integrated with Freetalk, and might have to be rewritten. Freetalk, WoT and all chat apps will have major scaling issues but hopefully these can be overcome. evanbd's Fritter microblogging app is a great idea, and Artefact2 is working on a blogging plugin which might be included in 0.8. Revocable SSKs (RSKs) are an important feature for hostile environments and will allow us to have an official project freesite. Filters for more content types are vital - particularly audio/video. Limited Javascript support, and untrusted/semi-trusted plugins, are possible given considerable work and various configurable security choices/tradeoffs. Even a form of video streaming may be possible. And swapping could be much faster and more secure. So there is a huge amount of work to do for Freenet to become a really awesome tool for both filesharing and censorship avoidance, and I hope that we may obtain further funding so that we can achieve most of it. To donate to FPI, click here, or to help us with code, translations, documentation etc, please contact us!

2009/10/16

Build 1237 and related things

1237 is now available, please upgrade. This fixes a bug in the client layer which was causing downloads not to complete (probably not the only one, sadly), fixes a minor exploit in the content filter, makes Library official and auto-loads it over Freenet, has more work on evanbd's hourly by-HTL stats logging, and some changes to the FCP feeds API.

Library is the new search plugin built by infinity0 and mikeb. It is integrated into the fproxy user interface, supports phrase searching, boolean operators, basic page ranking, and is a big improvement. But the real story is that Library supports the new Interdex search file format, which is an on-Freenet b-tree, which should be much more scalable than the current index format. In the medium term hopefully it will also support distributed WoT-based searching, which could form the basis for a really interesting filesharing system. The reason it's called Library is that it is not just the user interface: XMLSpider will be adapted to talk to Library to do its index writing. Hopefully in the not too distant future, the spider will use Library, and will write indexes much more quickly by bunching writes together in memory and then linearly rewriting all the indexes. We can do this with the old format, which would make it much easier to run a spider (as currently it can take 4 days to write a large index if you don't have a pair of Cheetah's or an Intel SSD), but it is even more interesting with the new, scalable format.

The content filter exploit was simply a failure to encode type parameters properly. The worst that it could do was allow a malicious freesite to inline lots of gigantic files as inline images/frames without querying the user about their size, but nonetheless it needed to be fixed. This was discovered during my work on merging the new CSS filter, which is more secure (because it parses CSS properly), and more functional (because it fully supports the spec, or will by the time it's merged). This will be merged within the next few days hopefully. Kurmi had done most of the work for his Summer of Code project, but there were various serious issues that needed sorting out. There is also an ATOM (but not RSS) filter to merge, a Thaw filter, and someone is working on an SVG filter (all of these should take much less time to review and merge). In the medium term, there is much more work to do: we need SVG in XHTML, XHTML in ATOM, possibly RSS, and hopefully some audio/video formats. Long term it'd be awesome if somebody could write a Javascript filter; this would have to replace all objects and functions that could be compromising with safe versions, it'd be a fairly big project, but it's quite possible for a good coder. None of the current devs know javascript well afaik, but maybe I'm wrong about that. To make eval() work, the filter itself would probably have to be written in javascript, or translated into it via GWT. Another interesting point: flash nowadays is essentially javascript with a different library and a binary format, so a javascript filter would go a long way towards flash support. Of course, on top of all this, you have all the worries about timing/datastore probing attacks etc, so scripting on Freenet is hard, but it's by no means impossible...

In the short term, evanbd's work on stats could turn out to be very important. Recent tests show that if you insert data on a well established node and fetch it immediately from another well established node, you can see transfer rates in the region of 40-80KB/sec, with one report of rather more. However if you wait 3 days, there is a disturbingly high failure rate - although that seems to have improved a bit since the last bunch of routing fixes. evanbd is working consistently on stats and simulations, and this has been very helpful, with the recent routing fixes coming out of it. Soon I will implement "tracer requests"; these will insert data using a special key type, record an identifier for the nodes it is inserted to, and then when we fetch them we will see where the request went to likewise. We will use a new node identifier that doesn't correlate with anything, and IMHO this is no more dangerous than probe requests (which we will be turning off), especially as it doesn't use "real" keys (we will create a very small datastore for such test keys). We think routing works pretty well, a popular theory for why data retention is so bad is that we don't handle opennet churn very well, but there are other possibilities, hopefully tracer requests will help us to understand what is going on. Given that Freenet can be fast, with some fixes and probably Bloom filter sharing, hopefully we can have a substantial performance improvement.

Other stuff since last post, which you probably already know about: the routing changes I mentioned above: for a while now we have taken into account not only our peers' locations but also our peers' peers' locations, but we were not handling this properly. The seednodes file is now auto-updated, and the Add a Friend page now serves the latest Windows and Java (OS/X and Linux) installers. We now have a Dutch translation, updates to French, German, Italian and Chinese, and more of the web interface can be translated. 1234 included some important work on memory usage, 1235 included some optimisations to the client layer (persistent downloads and uploads), trying to reduce database accesses and therefore disk i/o, cpu usage and memory. 1236 included some important work on ULPRs. Ultra-Lightweight Passive Requests are an optimisation introduced some time ago which is designed to make polling apps such as chat clients more efficient and faster, by remembering for an hour which nodes have requested a specific key, or we have requested the key from, and telling them if we find it. In 1236, we propagate ULPRs more effectively, more securely, and more accurately, as well as fixing a bug that may have been preventing it from working since July. Other recent work includes plugins, the datastore, the client layer, bandwidth limiting, node shutdown, automated tests and simulations, and more.

However, I have taken some time off work lately - partly because of minor ailments, but mostly because of getting involved in climate change events (all over until December for me now). Yesterday was the Blog Action Day, an attempt to have a co-ordinated campaign of blog entries about climate change; obviously I missed it! Unless strong action is taken to prevent severe climate change (defined as more than 2 degrees warming), we may find ourselves without "a planet similar to that on which civilisation developed", in James Hansen's words. Of course, something will be done - the question is whether enough will be done (Hansen and the small island states are arguing for a much tougher position than developed and rapidly developing nations are willing to commit to right now). The Met Office, for example, is saying that 4 degrees by 2060 is possible. Have a look at the New Scientist coverage if four degrees doesn't scare you. It won't be evenly distributed: some areas would have no warming, some would have 10 degrees or more. And as always, the poor are hit hardest, because they live in places which are already seeing severe changes in the weather, because they live off the land, and because they often have no resources to adapt with. Christian Aid estimates there may be a billion climate refugees by 2050 (others estimate 150-250 million), although some argue that those who are worst affected are probably also least able to migrate... The global poor are paying for our pollution, and the situation is going to get a lot worse even if we do take strong action. If we don't, vast areas of the world will likely become unfarmable due to drought, floods (often directly following drought), desertification, more natural disasters and so on. All of these are happening now, and have gotten significantly worse over the last 50 years, with only a 0.8 degree rise in temperatures since the industrial revolution. Maybe we'll be able to geo-engineer our way out of the most severe consequences, but it will be much cheaper and less painful to deal with the problem now. What must happen is drastic cuts in emissions - 40% from 1990 levels by 2020 in the rich countries, and some meaningful progress from the fast developing nations - and large amounts of cash to help the developing countries adapt and to grow in a low-carbon way (at least $150B). Neither of these demands is likely to be met fully in Copenhagen, with the US position constrained by domestic politics, India unwilling to take on any binding targets, and the Chinese so far only have vague mumblings about intensity targets, and demanding the right to keep on increasing emissions until 2050, by which time they will occupy the entire carbon budget for the world and then some (recent research shows they could peak by 2030 without any serious economic/poverty impact if they wanted to).

Sorting out this mess will be painful, energy prices will have to rise considerably and flying will probably in the long term become the preserve of the rich once again (but we'll have better rail), but in the near future a great many jobs could be created. Unfortunately many countries - particularly Britain - have squandered obvious opportunities for green recovery, preferring to spend vast sums on propping up retailers and car companies. Nonetheless there is a short-term economic case for action. The long-term case is rather more stark: the Stern review, years ago, said the economic cost of taking action would be around 1% of global GDP, and the cost of not taking action would be around 5%, but he also gave a worse-case estimate of costing 2% to take action and 20% to not take action. He has recently said that in the light of recent science the latter is probably more accurate. 20% of GDP is equivalent to two world wars and the Great Depression, and we need to start taking it that seriously!

So if you have not already contacted your representatives (national MPs and if appropriate MEPs), and told them that you care about the future of the planet, that Copenhagen must deliver, and that your nation must take drastic action both to lead in negotiations (40% by 2020, as Norway recently pledged, and their share of $150B+ of adaptation funding) and to build green infrastructure instead of brown infrastructure (Kingsnorth and Heathrow 3rd runway cancelled in the last month, yay!), please do so. There are many resources on the web and involved organisations e.g. 350.org, Stop Climate Chaos, Campaign against Climate Change, Operation Noah, Christian Aid. Read their resources and then email your representatives (UK) in your own words, because this is far more effective than clicking a standard form letter. After that, think about attending a relevant national demonstration - in the UK, The Wave is planned in London on the 5th of December. The talks themselves are on the 7th to the 18th of December in Copenhagen. The EU negotiating position will be finalised on 29-30th of October, so if you are in Europe you should write to your MEPs before then and maybe consider demonstrating during the session (although I'm not sure anything's organised yet, Climat et Justice Sociale Belgium may have something). The 24th of October is a global day of action for 350.org, there is probably an event near you, unless you are in the UK. In the UK, a colossal scheduling cock-up means there is a major anti-war protest (Iraq killed around a million people, climate change already kills that many every 3-4 years), the Anarchist Bookfair, and other local events going on... good luck, I'll probably attend the York Wave rather than try to organise anything in Scarborough. Recent events I've been involved in include Operation Noah's Climate Change Day of Prayer, for which I printed out far too many pages of printed resources (I suspect I'll be able to unload some of them though), and Tearfund's climate justice evenings (an equipping meeting that was interesting and encouraging), for which I had to sleep over in Leeds (one small disadvantage of living in Scarborough is transport is mediocre). I'll be in London for The Wave, and in Copenhagen for some of the period of the negotiations (if you are coming talk to New Life about accomodation, you probably won't find a hotel or a hostel). How much difference demonstrations outside international negotiating events make, when nobody can tell who are their constituents, is an open question, but it's good to have a presence, and the alternative summit should be fun. You can get an idea what is going on from the Adopt a Negotiator blog (on Freenet via Freerunner, RSS). Oh, and tomorrow Climate Camp and others will be shutting down Radcliffe on Soar, a major UK coal power station which produces more CO2 than a large group of least developed countries put together...

2009/09/04

Build 1233 and related things

Much work since last time! The XML bug, scaling max peers with bandwidth, plugin loading/update over Freenet, the minimalist theme and the new status bar, some minor filter changes with more to come, some important datastore fixes, more internal documentation (javadocs) and lots of bug fixes!

First off, the XML bug. This has been extensively covered elsewhere so I'll give you the basics and the current status: a remote code execution bug in a widely used XML parsing library is present in Sun Java up to 1.5.0_20 and 1.6.0_15 (amongst other things). Freenet will therefore refuse to load official plugins which parse XML on older versions of Java. 32-bit Windows users should be largely unaffected as the auto-updater will pick it up. 64-bit may have a problem as the auto-update only works for the 64-bit JVM and we need a 32-bit one (we are considering solutions given the increasing prevalence of Vista/64); download the latest 32-bit JVM and you'll be fine. Linux users may have to obtain the packages directly from repositories (e.g. the pool directory for debian), if your distro hasn't pushed the updated versions yet. OS/X 1.6 has the new JVMs, 1.5 update 5 will have it when it is out; currently Mac users have a big problem, but this will go away soon, so it is unclear that it is worth importing a separate library for it just for Mac users for a short period...

Scaling max peers with bandwidth: Freenet will now use between 10 and 35 peers depending on your bandwidth usage. This should improve speed on both slow and fast nodes (lower overheads for slow nodes, more routes for fast nodes), and it was the number one uservoice request for quite some time. Please let us know whether Freenet has improved performance-wise since 1231! Again this has been covered elsewhere. The number one item on uservoice is now "one GUI for all", which I interpret as a plea for more functionality integrated into the web interface...

Plugin updating over Freenet: We now load plugins over Freenet by default! Previously we were loading official plugins from emu directly, which is very hazardous and may not be possible at all in anywhere vaguely hostile, not to mention giving away your IP if you are on darknet. We didn't do this automatically - if a plugin was out of date, we'd ask the user whether to fetch it from emu - but it meant anyone unable to access checksums.freenetproject.org couldn't keep their plugins up to date - and the old versions frequently would stop working because of changes in the core. In 1233 for example there are core changes affecting 6 plugins, all of which will now be automatically pulled over Freenet on starting up; these fetches should work, but if they don't, Freenet will allow to either give up, let the node continue to try to fetch the plugins, or fetch them from emu. This is important now, especially for hostile regimes, but it is essential with Freetalk and Artefact2's upcoming blogging plugin!

Recently I've been doing a good deal of code reviewing and other miscellaneous stuff. This is great, because it means we have more volunteer work going on! Artefact2 has recently started contributing to the web interface (new theme, status bar), statistics (persistent uptime and bytes transferred stats), plugin infrastructure (localisation and data storage), and is working on a blogging plugin, which is something we should have had years ago. evanbd has suggested we implement a breakdown of request success rates by HTL, which has produced some remarkable results (have a look!), and promise to reduce the amount of alchemy that has crept in, largely because of the considerable difficulty of any objective measurements, since Ian declared the end of the age of alchemy all those years ago. He has analysed the data and helped with theory, and is working on a nano-blogging plugin and some changes to splitfiles. kurmi's XHTML and BMP filter changes have been merged, and his CSS, Thaw and ATOM filters will be merged soon. infinity0, Artefact2 and others have been working on a new freenet-ext.jar (please test and report back!) with non-segfaulting 64-bit FEC support, FreeBSD support, etc, please test it! Hopefully infinity0 and mikeb will be able to fix any remaining issues with Library so it can be the default new awesome search plugin (even if it may not have full distributed search yet), and more importantly, update XMLSpider to produce indexes (in the new format, supported by Library alongside the old one) much more quickly. Zero3's (and Juiceman's) new windows installer (please test and report back, no update.cmd at present, and not fully compatible with the old one) is nearly ready, but it has been nearly ready for some time so we'll see; it features a system tray icon, and doesn't create a custom user, so should have less of a problem with antivirus, system password policies etc. mrsteveman1 has had some trouble getting a systray icon to work on OS/X, but hopefully there will be one at least for some versions. On Linux, we will have to use a Java version, but I have been provided with some example code. saces has been working on various internal things and plugins, hopefully his multi-container freesite insert code will be made default before 0.8, it would be a huge improvement for big freesites. ljb's persistent node to node file transfers haven't hit fred-staging yet but we passed him (and the others) so I am confident they will. Our translators continue to be helpful, but as always we need more languages. There have been hints about Russian and Farsi but so far not much has actually happened. A possible Farsi translator said something about RTL support, we need to look into this but it might be easier with some help from the would-be translator; if you want to translate Freenet, please let us know. sashee's web-pushing branch is also approaching being merge-ready, and hopefully will be merged within the next month; loading the Activelink Index with pushing is quite impressive. Work on Freetalk (mostly by p0s aka xor) is slowly progressing, it should work in the current build but expect incompatible updates, bugs and so on. However I do read the Freetalk boards, so it may be the easiest way to contact me anonymously (Freemail is very temperamental for me). I've probably missed out a bunch of people and I'm sorry, I'm just trying to give a flavour of the current situation: There is a lot going on, most of it by new or re-engaged volunteers!

In the reasonably near future, changes to splitfiles (reinsert the top block when it takes more than a little while to fetch, split data evenly among a splitfile, make smallest segments a bit bigger) should improve data persistence somewhat, and if necessary MHKs will improve it further, without involving major changes. There are a few security and hostile environment matters that need to be dealt with, or that we would very much like to deal with: use more rounds in AES, encrypt plugins' on-disk data (and completed download list files), keep a copy of the latest installer on disk or at least link to it on the Add a Friend page, possibly RSKs (revocable SSKs, so if the private key is leaked you can still tell people this), some means to prioritise a modem-based darknet peer so it has some chance of its requests being answered, and possibly a Friend-to-Friend web proxy system to get around national firewalls with no frills attached (this is actually a lot easier than it sounds!). Freetalk is essential for 0.8, and hopefully p0s can get there with minimal help; an RSS filter would be nice to go with the ATOM filter, there are bugs in USKs but also there is the potential for major improvements, db4o auto-backups are essential, as is better usage of the datastore, and we also expect a few more usability/UI improvements (and I mentioned a few things mentioned already). Most of this is relatively easy. However, I still plan to implement Bloom filter sharing. All the indications are this would be a significant performance and data retrievability gain, and the initial implementation shouldn't take more than a couple of weeks, now that we have the prerequisites (caching changes) sorted.

Many other features have been postponed until 0.9, and many of the ones mentioned here may also be. Fortunately we have a reasonable amount of breathing space, as we have spent around a third of Google's $18K so far. So 0.8 should be really great! There have been a number of positive comments lately from users trying Freenet again after some years, and reviewing Content of Evil was quite educational, showing how far we have come. While on the one hand we will run out of funding sometime early next year, 0.8 could be a breakthrough in many ways, and I believe we will find additional funding - whether it be through paypal micro-payments from many concerned individuals (and I have been entirely paid off these at various points in the past, although I had lower costs then), from groups interested in hostile environment deployments (in some cases Freenet may be the best solution), or from some other source.

2009/07/30

Build 1226

Build 1226 is now out. A lot has happened in this build. Firstly, we have finally fixed The Register's attack. Basically the problem is freesites you browse, files you download etc were cached in your node's datastore, speeding up future accesses but making it possible either to look at your store and find out what you've been browsing, or to remotely probe your node's datastore (assuming the attacker is connected), and see how long it takes to fetch blocks to try to work out whether you have fetched particular files. What we do in 1226 is only cache in the datastore when the HTL is over some level, so your request will not be cached unless it is at least 2 hops away from the originator (often more).

Why has it taken so long? Well, we had assumed it would be fixed when we implemented premix routing or encrypted tunnels. These would also make a lot of other attacks a lot harder. Recently it has become clear that these will likely be a lot of work and may have a significant performance cost. One reason for these changes (which may reduce performance slightly) is that they are essential prerequisites for Bloom filter sharing, which should greatly improve performance, and will be implemented in the coming months (see more below). But another reason is that in many hostile environments, seizing a node and checking what people have been downloading is a very real threat, and up to 1226 Freenet left far too much evidence...

1226 allows you to configure the tradeoff between convenience, performance and security. Physical security levels now range from LOW to MAXIMUM. At LOW, nothing is encrypted by Freenet (but you might have whole drive encryption). At NORMAL, your node.db4o (downloads/uploads database) and your client cache (a new kind of datastore used to avoid fetching the same freesites repeatedly) are encrypted, with the keys stored in a file called master.keys. Securely delete that file and both are useless, and so are all the other temporary files as their keys are kept in node.db4o. The panic button has been fixed and reinstated, and will do this for you (but watch out for data-journaling filesystems and flash devices). On HIGH, you can set a password for master.keys, and you cannot access the download/upload queue until you enter this. On MAXIMUM, the keys are random on every startup, so there are no persistent requests, but you can still fetch big files as long as they complete before restarting.

Freenet migrates your node.db4o automatically on startup if it needs to be encrypted, it supports defragmenting node.db4o on restart (mine shrunk by a factor of 10), which it will do when you upgrade, there are security and performance improvements to ULPRs (ultra-lightweight persistent requests, used to speed up polling e.g. Frost, FMS), ljb's mostly internal work on user-events (did you know your node has an RSS feed on /feed/ ?), and bugfixes and optimisations. So that's 1226, please go get it! There may be a small performance hit in the short term...

The big performance feature for 0.8 is Bloom filter sharing. I have had a lot of help with working this out from evanbd, but originally it was Oskar's idea. Basically, we tell our peers what keys we have in our datastore (using a highly compressed data structure called a Bloom filter). This should cost around 300 bytes/second on a node with 30KB/sec upstream, but it enables us to short-cut and route directly to the node with the data when we happen to be nearby. For really popular data this means we can probably fetch it from one of our direct peers, and bypass load management. For less popular data it means we check 20 times more nodes' datastores, so once we are in the rough area where the data should be, we should find it efficiently. Hence it should improve performance, in terms of transfer rates, latency and how much data can actually be found, by a considerable amount, but we don't know by how much yet. Unfortunately we can't just deploy this for security reasons - hence the work above to ensure that the only stuff in our datastore is data which other peers at least 2 hops away have requested.

A big problem with Freenet is that it is relatively difficult to communicate with other users. This is partly because of the lack of a good integrated chat system, which p0s is working on with Freetalk. But it is also a fair amount of work, especially for less technical users, to set up jSite, build a site in HTML and upload it. It appears a basic blog engine can be written in 800 lines of php including templates ... so it might make sense to spend a few days on this as a relatively high priority, to make it easier for new users to contribute content. Freetalk would handle both comments and announcing a site. Granted, Thingamablog does something similar, but I don't think this has been maintained lately, and it is a separate application which you have to download; it should be a web interface, just like Blogger. This is not just a question of what will get us the most users (and hence funding) in the short term: If Freenet were to be used in a hostile environment (e.g. China), people using it would probably not be very technically literate, would be used to using web services to create content, and so on. Security matters, but usability matters too.

In other news, our Summer of Code students are doing really well this year. My student sashee has built a javascript push framework for Freenet, which in the web-pushing branch is used to update fproxy's site loading progress bars more quickly, smoothly and cross-platform (works on webkit unlike the current code), to update the downloads page on the fly, and to show progress for inline images while not blocking lots of browser connections (most browsers are limited to 8 or less to a single host). My other student, infinity0, is working on a distributed searching system called Interdex, which hopefully will transform both searching for files and searching for freesites. mikeb has done some great work on the existing search system (both spider and search plugin), ljb is working on friend to friend functionality (exchange of files, bookmarks etc between darknet peers), and kurmi has already written filters for BMPs, ATOM feeds and a new CSS filter, which will be integrated shortly. We will run out of funding in approximately January, but it should be a very interesting year!

2008/02/29

Chat and other matters

Freenet 0.7 is coming along nicely, although the tentative feature freeze is by no means absolute. Ian wants to release in March; a release candidate towards the end of March seems feasible at this point. ULPRs and related code (the cooldown queue) are pretty much finished, and there is a feature freeze, although some HTL changes have been necessary and one more may be in the next few days. Bugfixing is in full swing; the spider and the transport layer have been recent targets. We still have way too many timeouts, but many nodes in one VM simulations (freenet.node.simulator) have helped to find the cause of some...

However, one big piece isn't ready yet: a working, user friendly, hard to spam, bundleable chat client. The spammer seems to have taken a day off today, but until today he has relentlessly spammed all the default boards and many more popular boards, making them unusable. IMHO Freenet is a community, and for any community to function there must be a usable means for chat. Right now FMS is apparently working and under heavy development, but in C, and using an NNTP interface; seagull is apparently working on porting it to java, but we haven't seen his work yet. Ian is of the view that Frost and FMS are separate projects to the node and we shouldn't wait for them. What do you think? Answers to an appropriate forum, should we:

1. Unbundle Frost. We could just not ship Frost. This has the advantage that we could meet our deadline for shipping 0.7.0 more easily. And we wouldn't be shipping any known-vulnerable or known-broken software. But IMHO it would reduce user retention dramatically, at least by 50%.
2. Ship Frost as-is. If Frost isn't being actively DoS'ed, we could just hope that the spammer is a friend (e.g. an FMS advocate), and ship Frost anyway. If he isn't, he will DoS Frost as soon as we ship 0.7.0, even if he takes a break in the meantime. IMHO it would probably be a good idea to mention the fact that Frost has been actively DoS'ed and remains vulnerable in the 0.7.0 announcement: this will put off users, but if they have to discover it themselves, that will put off even more users. Our problem is not getting people to install Freenet, it's user retention: we have huge "join-leave churn".
3. Make it a release blocker and act accordingly. Review third party code, help with porting FMS to java if necessary, ship 0.7.0 with a bundled, working java port of FMS, probably with a web interface based on Worst. Have an official plugin (with anonymous contributors such as seagull, but also with non-anonymous contributors such as me) implementing the web of trust and exposing it via FCP, and another plugin (or an option in same one) implementing a web interface. You can get version 0.3 of the Worst source code here (if you have a more recent version then please post it to fms or to the fms board).

Note that because FMS is in C and therefore not bundleable, I don't currently run FMS. I may run the java port of FMS when it becomes available - hurry up seagull! I've seen test posts, so it can't be too far away...

In other news, we should seriously consider whether to take part in Google Summer of Code this year. I was approached on Wednesday by an enthusiastic and apparently competent would-be SoC student, who now has an SVN account and is working on a bug in the transport layer. Last year SoC was a lot of effort and a partial success - a lot of the code was of poor quality, but some of the devs have reappeared or not disappeared. If we do do SoC this year we will probably take fewer students, and we'll certainly want to be more careful in selecting the student (and not the proposal; proposals can be renegotiated).

2007/10/26

Radical update

A standard criticism of the Christian gospel is that the various resurrection accounts (bibles on freenet here or on the web here) are inconsistent. And at first glance they disagree on many important matters. I just found this gem (from the web), which makes a persuasive case that they can be pieced together into a coherent whole. Why does this matter? Well, if Jesus did rise from the dead, he probably is the Son of God. If he didn't, then Christianity (with all of its widely acknowledged moral contribution to the world) is nonsense. You can of course claim that the whole thing was made up, so it doesn't matter that they are consistent; but they were clearly written by different authors, and reasonably early although probably not direct eyewitness accounts; and Matthew's gospel was written to jews so claiming that the guards' tale was widely known among the jews probably would have been a bad idea if it wasn't true. Have a look!

I can't write a flog entry without a Freenet-related update. And I should probably explain what's going on locally too. Firstly, emu (the freenet project web/mail/etc server) has been down for a few hours, it will be back up soon, due to its being moved from London to Manchester to cut costs for bytemark; we're getting a free memory upgrade in the process. Secondly, we (family) are moving to Scarborough (wikipedia) on Thursday and may not be around for a while after (hopefully we will have a phone line when we arrive, so we should have dial-up, but there will obviously be lots of stuff going on e.g. unpacking). Reasons for the move include health (Bristol's air is really unhealthy), and closeness to relatives, and a much bigger house. Working from home has certain advantages!

Now for developments in Freenet itself: Build 1069 features a major security fix for our connection setup code: It was possible for an attacker to do a man-in-the-middle attack using weak diffie-hellman keys. Tor had a similar vulnerability in 2005, and Freenet 0.5 still has this problem (we are not going to fix it, we will apply a patch if you send in one, but Freenet 0.5 is unmaintained and unsupported). 1069 is mandatory on Tuesday, so upgrade! Hopefully the auto-updater will upgrade your node automatically, or is even now awaiting your clicking the "Update Now" button.

In other news, slow progress continues to be made towards full opennet. Path folding is working, UP&P (automatic port forwarding and IP detection if your router supports it) is working, reconnecting after reasonable downtime is working (I dropped my darknet peers, shut down my node for 12 hours, and it got back on its feet with opennet soon after starting back up; admittedly I'm not NATed...). 1070 will have a fairly major security fix relating to path folding (traffic analysis related to packet size ... sorry), and the major items remaining before we can implement automatic bootstrapping are automatic detection of port forwards (to get rid of the annoying connectivity messages and automatically detect if you are eligible to be a seednode), and anonymous connect crypto (the current link setup assumes both sides already know the other, which is great for darknet, and even for opennet, but not for bootstrapping opennet). So hopefully well before Christmas we will have full opennet support - full meaning you don't need to exchange noderefs with strangers, ever, even when you first connect to Freenet.

That's no excuse for running Freenet in its most insecure mode though! If you know somebody who has a freenet node, it is a really good idea to connect to them rather than to a total stranger chosen by the opennet logic. For several reasons: Firstly, your friend is likely to be more trustworthy than a random stranger, and Freenet is not perfect: those you are connected to can try to analyse your requests (see also the wiki). Secondly, it's not a random stranger: an attacker can choose to be connected to you relatively easily. Thirdly, your node is not invisible, an opennet node can be detected relatively easily, and then blocked on e.g. the Golden Shield (many western nations have some form of ISP-level blocking even now). And it is likely that one day Freenet will be illegal, even where you live, whether by the IPRED2 (pending, EU), the DADVSI (passed, France), or some other legislation targeting copyright infringers, terrorists and paedophiles (but most likely the first). Oh and you americans, you're not safe either: they've already tried for mandatory trusted computing once. Opennet is a transitional phase, a means to an end: Your node will automatically reduce the number of opennet peers (Strangers) it uses as you gain new darknet peers (Friends) (thanks to somebody on Frost for the idea). In the long run there will be no opennet. But in the short run it should allow much faster growth and a much easier install for most people trying Freenet.

Apart from that, there has been some other progress: JFK has been merged (new connection crypto setup, strongly resistant to denial of service attacks), there have been many bugfixes including some fairly juicy ones (continually polling USKs, node stopping sending requests due to stuck inserts, etc), better IP detection (we run the plugins in parallel as they can be slow), translation updates, major bugfixes and improvements to jSite (it is now able to insert large freesites without silently dropping files after the 1000th), XMLLibrarian, the freesite search engine, is working and has a default index (type XMLLibrarian# or XMLLibrarian* into the plugins box to load it via HTTP; the latter will reload it on every startup), you can now tell a darknet peer to drop you when you remove it, datastore performance fixes, faster HTML filtering, cruft removal, and much more, all since the last update.

But on the whole, progress is relatively slow. Why? Partly that life keeps getting in the way for me (don't worry about your donations, I charge FPI by the hour!), and largely because we have relatively few volunteer devs at the moment, although we just gained an anonymous freemail dev from Frost (Freemail is now a plugin, and appears to work at least sometimes: install it by loading Freemail* from the plugins page [ will reload from HTTP on every startup ], and send me a freemail!). Freenet is really a much bigger undertaking than the current team can handle in a reasonable time. If you want to help build a critical piece of a free online future, please contact us on IRC, the devl lists, or Frost. If you're interested in becoming our second full-time paid dev for Freenet, that might just be possible (no promises, sorry...), I advise you to start coding and contact us once you've established a reputation for being a capable coder (after say a month); that's what I did.

And the Google Summer of Code 2007 is well and truly over, I've sent a final round-up to the development list; we learned a lot from this year, and that was cemented by and added to at the conference Google hosted for us (which originally I went to because I didn't want nextgens to be the sole voice representing us! It was great though, much better than last year). Many thanks to Google for SoC, even though many of our students were disappointing; we got some useful code out of it, and at least two of the devs will hopefully stay on. Next year, if we are accepted, we will do things a little different!

Expect the next update when you see it!

2007/09/06

Something happening?

Google Summer of Code is finally over. Yay! Several of our students are likely to stay on as devs, and some useful code has come of it, although most of the projects were not fully achieved. Rough summary:

• Swati Goyal worked as my student on the XMLSpider and XMLLibrarian plugins. While there remains a lot of work to be done for these to be really good, both now work reasonably well. You can spider the freeweb with the spider, it will produce an index in myindex7/ every 10 minutes; copy that to a separate directory, insert it as a freesite. Then feed the URL you get into XMLLibrarian, and you can keyword search the Freesites it found. For obvious reasons the Freenet Project cannot maintain an official index, hopefully several anonymous persons will maintain indexes. You can group multiple indexes into a folder. Eventually we will have embedded search forms in freesites.
• Mladen Kolar worked as my student to develop a comprehensive yet simple to use FCP library in C++ using Boost - a library to make it easy for C++ developers to write Freenet clients. This seems to work, but it needs documentation and better example code; hopefully this will be dealt with soon. It may also eventually be wrapped in other languages.
• Vilhelm Verendel has been working on simulations over the summer. He's one of Oskar's students, and has provided valuable input on various aspects of routing and swapping. Most recently, after a paper confirmed what we had suspected for a while, vive's work enabled us to tweak the swapping algorithm to prevent the network degenerating into a state where the entire network is clustered around a small number of locations, either due to churn or malicious attack (some believe we have recently been attacked on this level...).
• Frederic Rechtenstein has built Echo, a blogging wizard/plugin for Freenet. I understand that this is very close to being usable, but it has some packaging problems. The objective is to make it as easy to create a blog on Freenet, in a completely decentralised way, under your control, as through any of the major commercial blog hosts.
• Alberto Bacchelli has been working on unit tests for many of the support classes that Freenet depends on. In the process he has found a number of bugs.
• Srivatsan Ravi has been building new link layer encryption code, which will be resistant to denial of service attacks. This will be merged when it is ready.

Apologies for the limited progress with Freenet recently: I never seem to finish anything, life has been rather mad lately, it'll be sorted out soon (after we move)... Current priority items:

• Opennet - Universal Plug and Play support for the opennet port, allow reconnection for a short period after downtime, announcements - we need to finish opennet ASAP, this will hopefully reduce the number of people who leave because they have to perpetually chase noderefs, although it's by no means a long term solution: you will all need true darknet peers eventually, because they will come and harvest the opennet and render you unto the people with the unusual definitions of torture (he resigned, yes!).
• Ultra-Lightweight Passive Requests - necessary for outbox polling (spam resistant) Frost, useful in general... started but not finished, current priority depends on whether progress is made on Frost...
• LRU not preserved on datastore recovery - we need to fix this, it causes loss of content (SSKs are lost too :( ).
• Probe traces - we are using this network-level debugging tool to explore the network topology and prototype new routing mechanisms, specifically the rabbit hole avoidance protocol. At the moment it isn't working. Arguably the rabbit hole avoidance isn't needed on a true opennet - but a hybrid network is bound to have pockets of nodes, dungeons, which are difficult to get out of once entered, and this is why we need to consider how to deal with this.
• SSK request flooding - it seems that we send a lot of SSK requests for ARKs and so on, to the point that the spider doesn't work if its priority is too low. This needs investigating.

Finally, I wasn't sure whether to post the following: some folk might get the wrong idea, and it does break my quasi-anonymous persona somewhat (not that I'm actually anonymous, of course..). But I promised to host it, so you'd have found it eventually, and more importantly, it might do some good: ritual humiliation (completely unrelated to Freenet (on the web).

2007/06/14

1037

Upgrade to 1037! Really! The long-awaited Update over Mandatory support will make life much easier for the developers when debugging the network, because nobody will be left behind by a mandatory build ever again (well at least until the next incompatible crypto/packet level change), and the new probe request code will allow us to test a new backtracking/HTL algorithm (which should help significantly on darknets and hybrid darknet/opennets) without breaking the network. Lots of other goodies too (less short packets, possibly less timeout, f2f file transfer...). And it'll be mandatory on tuesday. So go get it!

One reason for the short mandatory is that work on opennet will start in the very near future... We need opennet, because of topology issues (#freenet-refs sucks), and because we need more users, more developers, more money, more content. And we need it to grow in order to have enough users to be able to build a global darknet during the limited time remaining to us. So watch this space!

One last thing. On the resources page, you will find a full set of debian AMD64 ISO's (all 21 CD's). Let me know if you manage to fetch them! (Or even one of them...)

2007/05/25

Moving along

The original plan was to capture a load of data about the network topology and use it to determine what is wrong with the network and whether opennet is The Ultimate Solution™ (or just a necessary evil). In the process we discovered a routing bug, which was probably causing the topology data to be corrupt, as well as various more serious issues, and is fixed in 1033. It will be mandatory on June 5th, so that gives a window to fix some more bugs and implement some long-planned features in the meantime (not necessarily all of these todo items):

• Binary blobs - migration of your favourite freesites between disconnected darknet without knowing the privkeys. Required for next item. 75% complete as of this update, should be in 1035. Less than 1 day.
• Update over mandatory - updating your node from your peers, even if your node is so far out of date that it can't route requests to them. Will enable faster mandatory upgrade cycles when we are deploying new routing, load balancing code etc. 1-3 days.
• Datastore fixes - various bugfixes, but the main thing will be including all the data needed to reconstruct the index in the store files (right now if the index is corrupted we lose all SSK data and the LRU list). 1-2 days.
• New plugins - since several of our Summer of Code students will be using plugins, and since several third party projects are already using plugins, it's about time to sort out the mess that is the Freenet plugin API!
• New congestion control - Michael Rogers has built us a new transport layer architecture, thanks to extensive simulations and input from related protocols. This involves a new packet format and new congestion control code. And perhaps even a new load management layer (token passing).
• Multi-container freesite support - This must be implemented before releasing Freenet 0.7 IMHO. It would greatly improve the performance of larger freesites (both inserting and requesting). tar.bz2 support would improve it even more, but the only java version of tar I know about, Apache Commons Compress, has licensing issues (it's incompatible with both GPL2 and GPL3).
• Insert memory usage - Early this year a load of work was done on reducing the memory footprint of large requests. Similar changes haven't yet been done for inserts.

PS: Move your freesites! Build 1035, out Real Soon Now, will by default disable access to pre-1010 insecure SSKs (you can still enable access manually). In a month or two we will drop the back compatibility code which allows these (and old CHKs too) to be stored. What this means is you need to move your freesite now if it is still using an insecure SSK/USK key (one which ends in BAAE rather than CAAE, and for CHKs AAEC* rather than AAIC*). Or move your favourite unmaintained site (suggest coordination via Frost). I have moved the Greater Toad Pictures Stash.

One last thing: The first 11 CDs of Etch for AMD64 have been inserted on the resources page. Let me know if you manage to fetch any!

2007/04/28

Looking up for once... mostly!

For once things seem to be looking up. The network is, compared to recent history, zooming along, inserting a CD image in around 2 days, routing is highly specialised even on low bandwidth nodes, link crypto is secure (thanks to STS), we have 6 SoC students and I only have to mentor two of them, and FPI is in reasonable health due to Ian's newfound talent for finding rich donors. The next build will feature a full localisation infrastructure (this is 99% done, I just need to put a few more l10n keys in), so hopefully in the near future Freenet's web interface will work in your native language (contact us if you are a native speaker of a non-english language but also speak good english and want to translate for us, or validate other people's translations). As regards opennet, we are currently collecting data on the network topology, which should tell us whether #freenet-refs is having a major negative impact; if it is, we will commence work on opennet in the near future.

Now for the bad news. IPRED2 passed the European Parliament. This piece of legislation makes inciting, aiding and abetting, attempting or doing any intellectual property infringement intentionally (not defined) on a commercial scale (loosely defined) a criminal offence subject to full criminal sanctions (prison time, judicial winding up, denial of legal aid etc). In its original form this applied to all "intellectual property", which would probably have criminalised the entire EU software industry; fortunately the passed version excludes patents. Unfortunately, it would probably still make Freenet illegal under incitement or aiding and abetting. It is likely to be some time before this completes the EU process, and once it does it can be up to 18 months for it to be enacted in each state's laws. So we are on borrowed time. If you want to help Freenet out, now's the time!

In unrelated news, check out Ripple, it's a friend-to-friend darknet-style currency system (there is an old version which is fairly centralised but there is slow progress towards a true f2f system, I've had interesting discussions about routing with Ryan). The basic principle is that money is a system of IOUs: it's information plus trust. So make that system explicit, and instead of trusting the bank all the time, keep it as close to your social network as possible. It would be quite interesting for LETS style hour currencies, but even for hard cash it would be useful. Have a look.

2006/09/18

Crash

Backups are important. Not only that, but backups of the right files are important! I lost a hard disk... it turns out that my (weekly) backups didn't include my most recent GPG key (but I have the old ones), nor my Frost identity. They do now! On the upside, Servalan (aka amphibian.dyndns.org) now runs Debian Etch on amd64 (the workstation still runs 32-bit, to minimize disruption and because of openoffice and gaming complications). Now, how do you know I am still me?:

• I still have the keys to this freesite.
• I still control toad_ on irc.freenode.net (a registered nick with op rights)
• I have a page about this on amphibian.dyndns.org.
• I have signed the message announcing this with two older keys which you can find on the public keyservers, and with which I probably signed some messages on the lists in the distant past: here and here.

My new public key is E43DA450 in my keyring (4096-bit, yay!). The crash message is signed here with the new key. My new Frost ID is toad@zceUWxlSaHLmvEMnbr4RHnVfehA and my new freemail address is toad@freetoad.freemail (long version), although this isn't necessarily fully working/integrated yet; don't send any important correspondance over it yet, but if you want to send me a test message I'd appreciate it.

Geek computers are like the TARDIS. They are immensely powerful, but they're constantly breaking, because we're constantly tinkering with them! (Well in this case it was a case of abusing commodity hardware; why should a hard disk fail? I know even Seagate drives have a limited lifespan, but they shouldn't! :)).

On the other hand, last week's holiday (Monday to Thursday) was pretty cool. Me and mum went to see granny in Harrogate. Spent some time with granny, visited some nice green places, got lots of cheap videos from the charity shop, watched all four Harry Potter films, discovered that a cousin of mine is a Red Dwarf fan. Normal service (normal chaos?) will be resumed shortly; most of the TODO list on the last but one item still stands :(.

PS, here is the text of the Pope's original speech, if anyone cares; it's primarily about religion and rationality, but does contain a subtle attack on Islam even if we accept his apology with respect to the most blunt quote.

PPS, the below item (Hereticnet) has a Frost board called Hereticnet.

2006/08/29

Heretic

There are a number of persecuted groups who would greatly benefit from freenet's technology, but who cannot use it for moral or political reasons. For example, persecuted churches. Even if you are an atheist I hope you accept that freedom of thought, and therefore of religion, is important: You have the right to sincerely believe in the Jesus, Buddha or the Flying Spaghetti Monster, if you want to. Others have the right to ignore you and think you are crazy. So there may be room for a darknet variant which uses a lot of freenet's code, but has different goals. Note that I am saying nothing against Freenet itself: I like Freenet, I am morally happy with it, but I think there may be room for something else as well, once Freenet has reached a reasonable level of stability.

Such a network would be resistant to external censorship, but provide for internal censorship. In other words, it would be a high standards darknet: A community with its own standards for content, which it could enforce through expulsions and schisms, but which is not necessarily the same as the outside world's standard. On such a network, content inserts would be tagged with a cryptographic structure allowing the insert to be traced back one hop at a time, but only with the consent of (for example) 2/3rds of the nodes surrounding each hop. If somebody found some content they object to, they could file a complaint. This would be discussed on the chat system, and ultimately people on the network would inspect the disputed content (hence the need for a fairly 'high' standard), and decide whether to vote to trace the author, to trace the complainant, or to do nothing. If enough nodes vote to trace the author at each hop, he would be traced. He would then be identified to his direct peers, and everyone else would know his topological position. The network must then decide what to do with him. His direct peers may simply disconnect from him. Or they may choose to protect him, (either after the trace or during it), in which case they themselves may be disconnected from. Irreconcilable differences will have to be dealt with by a larger network split: What was one community is now two.

This is by no means an easy way out of the conundrum that is freedom of speech. It requires significant effort on the part of the users, and it also requires a fairly high standard; "anything but child porn", for example, is likely to result in permanent brain damage (or at least a need for counselling) to active participants on the network, since disputed content will normally be close to the border between what is allowed and what is not. A persecuted church would have a much higher standard, while most of its content would still be illegal by the local laws. And it is likely that such networks would have major problems with splits and schisms, as any other community does. It would closely represent the underlying community. It seems to me that this would be an interesting experiment, and it might be useful to somebody. Comments? Contact me!

2006/08/19

Rolling on

Probably a good idea to update my freesite once in a blue moon, right? Well there have been lots of goings on in Freenet land... Minor stuff first. Build 950 somewhat experimentally makes the node treat backoff as advisory rather than mandatory, so your node will send requests to backed off nodes if it can't send them to nodes which aren't. This seems reasonable to me, (because we have a separate load limiting system), Anyway it's not a long term solution; the current mechanism has major security issues (flooding and possibly some local attacks too). Tell me what you think of performance on Frost. There have been many other improvements, a good deal of which is due to our volunteers and Google SoC students (students Google pays to hack on freenet over the summer); upgrade! As you've probably heard, John Gilmore (who remarkably enough owns toad.com) has donated $15,000 to Freenet, so I'm definitely going to be paid for the next six months. Obviously this is a good thing!

One of the big questions in 0.7 now is opennet. Here's the deal. On darknet, you only connect to your friends (in theory; in practice most links are established via #freenet-refs etc between strangers). On opennet, the network assigns you connections. Either way, you are vulnerable to your peers: they can do correlation attacks, and probably lots more fun things. On opennet, you are probably connected to the Bad Guys, because they probably harvest the network (impossible on darknet), pretend to be a thousand nodes (hard on darknet), and connect to everyone they can - preferably more than once under different identities. In other words, Opennet is vulnerable to harvesting (quickly finding large numbers of nodes) and Sybil attacks. For the time being there isn't much we can do about correlation attacks; I have a few ideas which may be implemented soon but they won't provide a very large anonymity set; expect major progress in 0.8, but it's quite likely that premix routing won't work on opennet. So as you can see, we need opennet for bootstrapping: for getting slashdotters onto the network. In the long term, we need as many true darknet connections as possible, because Freenet will be illegal or attacked. One thing which is important is not to jump in too soon; many problems are easier to solve on darknet, which has lower connection churn etc. But we do need opennet, and once we have it we need people to move from all opennet to some opennet and some true darknet connections, and eventually to pure darknet, or the network will be unsustainable. What incentives we can provide is the subject of some debate.

So when will Freenet be illegal? The DADVSI in France arguably makes it illegal there. We may have a constitutional case, or we may not, but from what I've heard we are in trouble. People who distribute or develop it may be liable to prison time and a €300,000 fine. What's more fun than the DADVSI then? The IPRED2 is a directive going through the European Parliament which makes it a crime, punishable by all manner of criminal sanctions including prison time, judicial winding up, large fines, and lots of other fun things, to intentionally commit, attempt, aid, abet, or incite (!) the violation of any intellectual property on a commercial scale. "On a commercial scale" is undefined, but caselaw suggests it's not a real protection for anything either Freenet or the free software movement does; "intentional" may not require knowledge of the specific IP according to the FFII. But even if it does, knowingly violating a software patent on the grounds that it's a load of nonsense would now become a criminal offence! The application to patents alone will seriously harm the free software movement, of which Freenet is a part, and fighting it on the patents issue is the best way to attack the directive (as we won the fight last year). But the application to copyrights might well ban Freenet EU-wide. Please don't get depressed and cynical about this. Join the FFII, read as much as you can, and write to your MEPs. Last year we fought the proponents of software patents (including IBM, which is either schizophrenic or trying to have its cake and eat it, or both) to a stalemate: All that is required for evil to triumph is that good men do nothing, but there is much that we can do (before we reach the "we must all hang together..." stage!). Here's a letter (HTML) I sent recently (addressed to Lord Sainsbury, although you should generally write to your MEPs on this); please don't send exactly the same letter, but you're more than welcome to use it in writing your own.

Finally, the government expects all UK broadband ISPs to be blocking child porn sites using Cleanfeed by the end of 2007. They have assured me that it will only be used for this purpose, but I still worry that ISPs will be forced by litigation to block specific copyrighted content (e.g. xenu.org) - certainly they will if the IPRED2 passes. We shall see...

TODO LIST: Current relatively high priority items:

• Multi-container freesite support - currently any content in a freesite over the 2MB container limit is inserted separately; multi-container support will help a lot with big sites.
• Station-to-station protocol - Freenet 0.7's crypto at the moment is vulnerable to both spoofing and MITM.
• Low level rewrite - Michael Rogers and I have been debating congestion control, bandwidth limiting, load limiting, and a little bit of encryption for some time, while he has been writing simulations of these for his Google SoC project. A lot of this could be put into practice in the near future, although it'll be a while before token passing load limiting is ready (but when it is it will solve lots of problems).
• Location probe requests - We have some persistent suspicions that location swapping is causing large numbers of nodes to have locations in roughly the same place, and sparse keyspace regions populated only by newbie nodes which don't last long. Location probe requests would let us investigate this, and also get an accurate size estimate for the network.
• Freemail and SNMP - SNMP has been working for some time, and I want some pretty graphs! Freemail (Dave Baker's Google SoC project) is now entering final testing. Hopefully next time I post I can include a freemail address!
• Other security issues - There are some nasty security tradeoffs in 0.7 (e.g. whether to cache local requests), I have some ideas for improving security without too much work (the big gains are in 0.8 where we get premix routing etc).
• General bugfixing etc - Many minor but important features aren't yet implemented, and there are always more bugs...
• And of course, OPENNET !

2006/06/02

Shadows of shadows

A freesite with a few links to stuff downloaded from a certain p2p program recently came to my attention. It contains at least one file which is pure copyright infringement, so I won't link it here. But it also contains this file, labelled on the original site "US ARMY 3 - top secret defense - Biological Weapons Technology.pdf". Sorry to burst your bubble, but it's on the web. Just google for "Those delivering BW could be". That gets you to www.fas.org/irp/threat/mctl98-2/p2sec03.pdf. Go up a few levels and you find all sorts of wonderful stuff about WMD and intelligence threats and so on. Another few clicks and you get to stinet.dtic.mil. An amazing amount of this stuff really is just out there, and there is absolutely no shame in pruning through it and finding the good stuff. Or even calling attention to it. I honestly don't know what the legal situation is with this sort of stuff, but frankly I don't care; it's almost certainly covered by copyright exceptions in my country. Have fun, and insert anything you find especially interesting!

Welcome to Freenet 0.7!

I have finally updated my Freesite, two years after the last post - and this time it's on the long-promised Freenet 0.7 alpha darknet! Yay! here is a page about Freenet 0.7's current security or lack thereof, from the wiki. Also, we are looking for student coders to work with us for money over the summer, thanks to Google's summer of code. You will however need to blow your anonymity by visiting the web site to find out about this.

Naphtala

The excellent (if small as yet) new index called FreeHoo carries in-depth reviews of Freesites. His second site listed was Naphtala, a site which I have never visited because I believed it to contain child porn. It doesn't. It contains a fair amount of interesting content written by a paedophile, none of which qualifies as pornographic or illegal as far as I can see. He claims at one point to use non-violent child porn so that he doesn't have sex with real girls, while consistently defending his practice of "child love". Definitely worth a read, as Freesites go, but I won't link it here directly for purely paranoid reasons. Obviously I do not endorse his views: any sexual relationship with a child is evil, illegal, a sin, and likely to damage them for life (I go much further than most people here by endorsing the biblical view of sex, which is that it is a good thing, within marriage). However it is interesting to hear from the other side, and I rather pity Naphtala.

Brain Dump - why do I do Freenet?

I was recently bothered by the whole thing of why I do Freenet, and whether it is compatible with my ethical basis. I include the resulting braindump in case anyone cares. You may find it interesting. I rather hope this won't result in me becoming even more of a cult figure, but perhaps it will spread some useful insight.

Why do I think it is right for me to do Freenet?

As an evangelical Christian, I'm not the stereotypical Freenet user. I don't surf the porn sites, I accept an ultimate (although not human) Authority, and so on. I don't define myself as a Libertarian or Anarchist. Many of my brothers and sisters in Christ would probably not approve of what I am doing. So, why do I think it is appropriate for me to do Freenet?

Firstly, Why is this difficult? Well, firstly, my income comes from working on Freenet. I don't have any real financial commitments to worry about, but it is better that I am able to contribute to the family running costs. And I enjoy giving to both Tear Fund (an explicitly Christian overseas charity operating in Africa, Columbia and other "nice" places; they help people to work their way out of poverty, provide emergency relief, and witness the gospel in both their words and actions) and my local church. Thus I have a vested interest here, which makes objective ethical decisions difficult.

So, now the meat. I do Freenet because I believe that freedom of speech is, despite not being a biblical concept, something worth fighting for in a democracy. Does this mean I idolize it? If the state has the right to taxes ("give to caesar what is caesar's...") and even to wield the sword (one of Paul's letters), then isn't it legitimate for it to say what people can and can't say? It is only a small step from here to "Isn't it legitimate for the state to say what people can and can't believe? Isn't it within the bounds of statehood for Rome to persecute the early Christians?". If we look at it from this perspective... Rome persecuted the early church, and failed. Badly. The church did not fight Rome (it was not a good idea historically, nor was it compatible with the sermon on the mount). But the Church prevailed, and those martyred (in horrible ways) will be rewarded in heaven for their perseverence.

So how does this relate to Freenet? Well, Freenet can be used for good or evil; it could be used by Chinese student groups to keep in contact with each other and the West (for example to organize prayer when they are particularly persecuted), by urban Saudi churches to keep in contact with current Western christian literature, by Western whistleblowers to publish things such as the Diebold files, or to publish criticism of the Church of Scientology, by paedophiles to exchange pictures of child abuse, by others to publish racist hate speech, and by yet others to publish lists of people accused (but never proved.. or who have served their time) of paedophilia and related offences, for future ostracisation and lynchings. Surely it is the state's right to prevent the last 3 cases?

Firstly, even if there was no Freenet, there would still be files that while technically illegal are clearly in the public interest, and should be hosted by as many people as possible. For example, the various sites on the Church of Scientology (read them, they're on Freenet). These are of clear benefit to the kingdom of heaven. The state's regulatory role is trumped by the clear good that is being done in keeping people away from this vile cult.

I have always thought that what Paul said about the state's right to wield arms, and so on, are guidelines - true in general, but occasionally it may be necessary to go up against the state; to disobey, in extremis to actively work against the state, perhaps using violence as a last resort. The resistance against the Nazis in WWII is one clear example of this. If you have Jews in a cellar, and the gestapo come and ask you about them, it is perfectly okay to lie to them. If a higher cause is being served it may sometimes be necessary to steal or even to kill, in such extreme circumstances. However on the whole, people lie to cover up sin, or to "fit in" (fear of man"), people steal to feed their sinful desires, and people kill for much the same reason.

So where does that leave us? We have to fend for ourselves, and work out exactly what the costs and benefits are. Even if the state has a legitimate right to suppress lists of people accused of crimes they were never prosecuted for (for obvious reasons), that does not mean that it will be successful. Very often such people are persecuted - often in error - because somebody in the police themselves tipped off the locals, or the paper. Once this information is available, it will spread, because it is believed to be in the public interest. Sometimes perhaps it is. If such data is published on Freenet, and the author is believed (which may be difficult - he needs to earn some trust - but this is certainly not impossible - and in this particular case it may not be hard :( ), then it is difficult for the state to get rid of it. To that extent, our building of Freenet has helped this to happen, even if we do not ourselves approve of such content's publication. Similarly in the case of paedophilia: we hide the people distributing the images, and although the police may well be able to investigate in other ways (e.g. matching the children involved), they are not able to simply trace the distributor, unless he does something stupid to give away his identity (trapping him on a message board may work). They are therefore available to a slightly larger audience at a somewhat reduced risk (although theoretically if they are not accessed they SHOULD fall out of Freenet).

So we are clearly building something that CAN be abused. As can a kitchen knife, or just about anything else, but more specifically, weaponry. We are building a weapon here, of a sort. Like all weapons it is not tactically neutral; it cannot be used to suppress information. It can be used to prevent such suppression. My intent is that it be used for the positive purposes listed above, and more. But some will inevitably use it for evil. How do I know that the good will outweigh the evil? Lets look at the good first:

Freenet can probably be shut down relatively easily in its current state by a moderately well funded attacker. For example, a half time Chinese technician (of some skill), with access to the firewall rules, could set up some nodes, harvest some references, and also the seednodes, and block every node found by IP address. However, compare it with what they have now (or did before recent developments): a mailing list on which open proxies are announced. Every 24 hours or less, a new open proxy is found and announced. The government is subscribed to that list, and within 24 hours, the new proxy has been blocked. Using Freenet (or I2P, which has similar attack issues) instead makes the cost to the Chinese government higher. If Freenet gets bigger, it becomes harder for them to block, but it remains reasonably feasible. However, should Freenet routing start to work, we may be able to provide new features that make it substantially more difficult: trusted mesh routing, which would severely limit harvesting in hostile environments, and thus limit the damage from busting one node; new (steganographic) transports, which slow down and make more difficult attacks, and so on. In fact, right now, you can identify all Freenet traffic by some identifier bytes. But we will get rid of this. And even if Freenet is never really THAT useful, there may well be future anonymous P2Ps that are. We will have laid the groundwork, even if they don't use any of our code, through our extensive practical research. Little of it is formally published, but I still think a lot of that knowledge would be out there.

In terms of impact: revolutions are always (in China anyway) led by the students. Even in China, the students have internet access. Bloodless revolutions have happened on many occasions in the last century. But the people need to know that another world is possible in order to demand change. I do think that such things *ARE* God-blessed. Velvet revolutions as we saw in the Soviet Union, in Serbia, in Georgia, are usually surrounded by a delicate chain of coincidences that is the signature of divine intervention; God *is* interested, and he *does* care. And the stakes are pretty high: there is a good chance of China and the USA having another cold war, quite apart from human rights issues, and the persecution of the Church in China.

In the West, files such as the Diebold files (leaked info on voting machines with major security issues), the Church of Scientology files (which the Co$ has consistently abused copyright law to pursue, even if they are readily available *NOW*), and so on, demonstrate that freedom of publication is vital to protecting democracy. And democracy has achieved great things for us, even if it is constantly under attack. At its best it can be a great means for change in a positive direction. At its worst, it devolves into plutocracy, as do all other governmental systems. The basic difference is the attitude of the electorate. If they are apathetic, evil triumphs; "All that is required for evil to triumph is that good men do nothing", as a wise american once said. So as always, true progress comes from the renewing of minds; but nonetheless democracy is valuable and worthwhile. And without freedom of speech, democracies devolve, as the public do not know what is going on, even if they DO care.

In the future, the West may become a very dark place. I cannot know for certain but I suspect we are in the twilight of democracy here. The War on Terror expands, and if it is not over quickly, it risks destroying democracy. If there is another serious attack, we will take another major step into the abyss. The intelligence war may be going well, however, there is plenty that the terrorists could do, and the war for hearts and minds is largely ignored (as we see with the reluctance of the US to bring real pressure to bear on Israel). If it continues (and it may, as there is considerable money to be made on both sides), we are going down the tubes - and this can't easily be fought with the prospect of terrorists crashing jets into nuclear reactors on the horizon. The other big influcence is in cyberspace, which increasingly dominates meatspace as a communications medium. It is quite possible that the little war between open source software and Microsoft et al will come to an end. If Microsoft loses, its stock price will collapse, and this will have substantial effects on the US economy (IIRC Microsoft is 20% of the S&P index, which a lot of pensions are linked to; not sure about the NASDAQ). If Microsoft wins, we will have increasingly unreasonable DRM regimes, and the War on Piracy will start to move. Because there are always bugs, automatic updates of trusted operating systems are inevitable. Because you cannot prevent people from aiming a camera at a TV screen, we may well be looking at a means of remotely deleting files from all trusted PCs everywhere that match a particular checksum. Once this is technically available, even if judicially supervised, it will be abused. Eventually it will be abused for overt political goals, but where there is a blur between copyright and freedom of speech, the former will prevail, as it usually has in the past, and increasingly will in the future as further legislation passes which makes it ever more obvious which side the legislators' bread is buttered on. Freenet will not be the only key piece in this fight; black-market hardware may well be an issue, and there will be many more; and I personally will not be involved, once OSS (or Freenet) is made illegal, because of my prominence beforehand; but this is another area where Freenet and similar technologies may well be of some practical benefit. And the Church will not be unharmed either. Prohibiting radical islam would be politically incorrect, so what is likely to happen would be prohibiting religious intolerance (there are already laws about this, we will have to wait and see how they are interpreted). Sermons and literature on why other religions are wrong, or why homosexuality is wrong, and so on, would become risky. If the WoT continues to escalate, or if there is a regional nuclear war (as the world continues to "tool up"), we may end up with explicit regulation of religion.

So, in conclusion: I do not know whether Freenet will be a net benefit. I do know that I am writing a tool that can be used for good or evil, that the good it can be used for is considerable, and the evil that it can be used for is there. God, however, knows. So it all comes down to whether God is happy about it. Which is expressed in my conscience; the indwelling Holy Spirit informs a believer's conscience. And I don't really have a problem here, most of the time, despite having a better relationship with God than I had some years ago. It is also expressed practically. I pray to God for success, and I have success, often from unexpected quarters. I have had huge last minute donations; I have been led to enormous bugs by coincidence and hunches after prayer. I have been blessed in many ways. I also honestly believe that the project we are undertaking is massively difficult, and a lot comes down to luck; we do not have the resources for it to be otherwise. I don't believe in luck; I believe in providence. I have met several Christians through Freenet, despite the natural expectation that everyone involved in such a project would be either very shallow or a committed Libertarian atheist. And so I continue. I believe, and trust, that if it is wrong for me to do Freenet, God will reveal this, in time. And even if he does not, or if he makes it abundantly clear and I ignore it for years, I will be forgiven at the end of time, because He has paid all my debts, and gone where I could not return from, and come back, and given me the promise of eternal life, secured by the only true authority in the Universe. Glory be to the Lamb forever and ever!

Hutton Craziness An interesting puzzle...

The Hutton Report (Freenet) was published today. I haven't read it (been far too busy with Freenet), but the general gist of it, according to the press, is that the government has been completely exonerated. A lot of other interesting stuff came out during the process of the inquiry, but the conclusion is interesting, given the fact that we still haven't found any WMD in Iraq, and Hutton says the dossier (Freenet) was _not_ 'sexed up', we have a few options:

1. Hutton is wrong This seems to me to be the most credible option: the government did indeed produce a misleading bunch of factoids, either completely independant of any actual intelligence, or exaggerated and carefully selected to look good. This could be a more subtle long term problem: agencies, individuals, sources that produce good intelligence are rewarded, but there is no way to measure how "good" intelligence is, so you check it against your political goals and preconceptions...
2. There is a pig flying the plane If Hutton is right, then the intelligence services did indeed conclude that Iraq had Weapons of Mass Destruction, without prompting from the government. If this is the case, given that we haven't found any, we have some more options:
   1. Iraq really does have WMD We just haven't found it yet. This seems increasingly unlikely as time goes on. And if it does, they are highly unlikely to be deployable, since they weren't used in the war, and they haven't been found since despite the disbanding of the Iraqi army and totally unrestricted access for the team of 1,400 US investigators to all sites in Iraq.
   2. Our intelligence services are incompetent If Iraq does not have WMD, but the intelligence services believed impartially that it does, then logically a third world nation must have deceived the CIA, the NSA, GCHQ and SIS, DIS, and all the other three letter agencies involved in many countries. The US intelligence budget is rumoured to have been $28,000,000,000 in the late 90s and is undoubtely much higher now.
   3. Our intelligence services are malicious The other option is that they did not seriously think Iraq had WMD. If they then lied to the government, as opposed to the government lying to us, then they are completely out of control, and presumably had their own geo-political agenda, perhaps something about the 7 million barrels a day Iraq could produce if its oil reserves were properly explored, and what that could do for western economies. Or something less subtle... Of course this doesn't have to be a global conspiracy - presumably if the american agencies wanted to believe in Iraq having WMD, they could persuade the UK agencies of this fiction. Of course, if this is true, sooner or later something may be planted to sate the public...

The other interesting part is that no British government is going to go to launch a preemptive war on the basis of intelligence for the foreseeable future. This is probably the real payoff here. America loses its biggest ally in the fight against States We Don't Like. How sad. Sidenote: I actually think Iraq is better off for the war for humanitarian reasons. That's not really the issue here though.

And now for something completely different: Freenet Update Freenet is currently having some major problems. Both branches are running NGRouting, which could probably be improved significantly; m0davis is working on that, when he's around, which is rare. The unstable network probably has 100 nodes, and seems to work remarkably well for new content, with insert/retrieve tests succeeding immediately even at relatively low HTLs, most of the time. Stable however isn't. Stable is estimated to have on the order of 10,000 nodes, as of a few months ago: Iakin ran an ubernode for 4 days and had 16,000 unique IPs contact it. The maximum HTL has been reduced to 10 on the stable network to try to reduce load. At least on stable, nodes are in overload almost all the time, and it's almost all the time caused by bandwidth usage. However, this is expected; we now reject queries based on outbound bandwidth usage, with the result that when we come out of overload, if we get a few queries for files that are actually retrievable (and yes, this does happen), our bandwidth gets used up and we go back into overload. After a somewhat paranoid conversation with an old friend, I discovered a possible attack that could have caused the current symptoms, you can read the thread here, which was fixed in 5064, but not completely resolved, as you can read in the thread. This may be happening a little by accident, but I doubt that it's happening a lot by accident, because the log messages that would show it are relatively rare ("Got really late DataReply" is the obvious one). Anyway, 5064 fixes it, but it also makes probing the network for a given key a bit easier, so there will need to be further action on it. I am currently engaged in implementing a new load balancing system, based on the idea of enforced, explicit maximum request rates, which was originally proposed by Ian. The result should be that the load is moved back to the flooder - if a node makes too many requests, it will get RNFs rather than overwhelming the network. This should help to balance load and heal routing... but we've all heard that before. It also gives us some interesting possibilities w.r.t. fairness - we can change the minimum request interval on a per node basis, so that we can "punish" nodes for bad behaviour (such as the attack above), or accept more requests from nodes whose queries are most likely to be successful. This should not take more than a week to implement, and it will require a network reset. After that, we are going to attempt to set up a "testnet", an expanded form of the old watchme network - a completely separate, non-anonymous network for testing routing, debugging, and so on. Unfortunately for it to be of any use we will need several hundred nodes to run it. But eventually it should give us a much better idea of what is going on.

Rants go here! I replied to jrand0m's last post on the list IIRC, but future ones will be replied to in both places. I'm sure I'll have other things to rant about occasionally.

Since jrand0m asked nicely, and since I knew he wouldn't mind me borrowing his template, I've decided to set up a rant site of my own.

I actually got a mail from a Chinese citizen on the support list today. He can't get to the web site, as it is blocked. Hopefully I can send him a ZIP. I still think anyone using Freenet in China is crazier than I am though - but as Ian often points out, what they used before was even worse.